arXiv: 1506.03148vl [math.AP] 10Jun2015 


On a doubly critical Schrddinger system in 
with steep potential wells* 

Yuanze Wu“ &; Wenming Zou^ 

“ College of Sciences, China University of Mining and Technology, 

Xuzhou 221116, P.R. China 

^Department of Mathematical Sciences, Tsinghua University, Beijing 100084, P-R- China 


Abstract: Study the following two-component elliptic system 

{ An — (Aa(x) -t- ao)u -h -t- = 0 in 

Av — {Xb{x) -h bo)v -h + Pu^v = 0 in 
{u,v) e X 
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trivial solutions is also unknown in the literature. We observe some concen¬ 
tration behaviors of ground state solutions and general ground state solutions. 
The phenomenon of phase separations is also excepted. It seems that this is 
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the Sobolev critical exponent. 
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1 Introduction 


We study the following two-component elliptic system 

{ Au — {Xa{x) + ao)u -I- -I- f3v‘^u = 0, x in 

Av — {Xb{x) + bo)v + + Pu^v = 0, x in (’^a./s) 

(m,x) e X 

where oq, &o G R are constants and A > 0, /3 S M are parameters. The potentials 
a(x) and b(x) satisfy some conditions to be specified later. 

It is well known that the solutions of iV\^p) are related to the solitary wave 
solutions to the following two-component system of nonlinear Schrodinger equa¬ 
tions 

- - Aa(x)^'i -f -h /3|«'2p«'i = 0, 

d 

< -i —^-2 = A«'2-A6(x)^'2-h|«'2p«'2+/3|«'lP^2=0, 
xinR"‘,t>0; Tj = x) G C, j = l,2, 

, d'j(t,x) —^ 0 as |x| —>■ -1-00, t > 0. 

Indeed, set 'I'i(t,x) = e“**““M(x) and 'I' 2 (t,x) = e“*‘^“?;(x), then ('I'i,'I' 2 ) is 
called the solitary wave solution of and {u,v) is a solution of the {V\,p) 

if and only if (di'i, ^' 2 ) is a solution of the {V^ p). 

In the literature, the System p) defined on an open set Q, (in K.^ or 
R^) is called the Gross-Pitaevskii equations (e.g. [26l |45]), which appears in 
many different physical problems. For example, in the Hartree-Fock theory, the 
Gross-Pitaevskii equations can be used to describe a binary mixture of Bose- 
Einstein condensates in two different hyperfine states |1) and |2) (cf. [H]). The 
solutions 4'j(j = 1,2) are the corresponding condensate amplitudes and /3 is the 
interaction of the states |1) and |2). The interaction is attractive if /3 > 0 and 
repulsive if /3 < 0. When the interaction is repulsive, it is expected that the 
phenomenon of phase separation will happen, that is, the two components of the 
system tend to separate in different regions as the interaction tends to infinity. 
The Gross-Pitaevskii equation also arises in nonlinear optics (cf. [T]). Due to 
the important application in physics, the Gross-Pitaevskii equation {Vq p) has 
been studied extensively in the last decades. We refer the readers to [5j [181 EH 
|30l|35l|37] and the references therein, where various existence theorems of the 
solitary wave solutions were established. 


When we consider the equation {V^ p) or in R^, the cubic nonlinear¬ 

ities and the couple terms are all of critical growth, since the Sobolev critical 
exponent 2* := 2N/{N — 2) = 4 in R^ = R"'. By the Pohozaev identity, we can 
easily conclude that any solution of {'Po,^) satisfies /g 4 uqu'^ + bov^dx = 0 (cf. 
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[min]). Thus, any solution of (Po./s) niust be (0,0) in the case of oq^o > 0. 
Due to this reason, to some extent, it seems that A 0 is a necessary condition 
for the existence of non-zero or even non-trivial solutions to {Vx^p). 

Definition 1.1 We call that {u,v) G x is a non-zero solu¬ 

tion of {V\^p) if {u,v) is a solution of {Vx^p) with {u,v) ^ (0,0); we say 
{u,v) G i7^(]R^) X is a non-trivial solution of {Vx,p) if {u,v) is a non¬ 

zero solution with both u ^ 0 and v 0. 

To the best of our knowledge, few result has been established for the Sys¬ 
tem {Vx,p)- In this paper, we will study the System {Vx,p) with A > 0 when 
a{x),b{x) satisfy the following conditions: 

(Di) a{x),b{x) G C(M^) and a{x),b{x) > 0 on 

(D 2 ) There exist Coo, 600 G (0, -foo) such that lim a{x) = Ooo and a{x) < Ooo 

\x\—^-\-co 

for all a; G R^ while lim b{x) = boo and b{x) < boo for all a; G R^. 

|x|—)-+oo 

(D3) D(j := int a“^(0) and D;, := int 6“^(0) are bounded non-empty domains 
and have smooth boundaries. Moreover, Qa = a“^(0), fib = ^”^(0) and 
Da nUb = 0. 

In the sequel, Aa(x) and Xb(x) are called the steep potential wells under the 
conditions {Di)-{D 3 ) if the parameter A is sufficiently large. The depth of the 
wells is controlled by the parameter A. An interesting phenomenon for this kind 
of Schrodinger equations is that, one can expect to find the solutions which are 
concentrated at the bottom of the wells as the depth goes to infinity. Due to 
this interesting property, such a topic for the scalar Schrodinger equations was 
studied extensively in the last decades. We refer the readers to H HI HOI US 
[SmSHlIlB] and the references therein. Most of the papers are devoted to the 
subcritical case. In recent years, the steep potential wells were also introduced 
to some other elliptic equations and systems, see for example [Miiiaiiziiiiiiin] 
and the references therein. In particular, in [49], the Gross-Pitaevskii equations 
in R^ (subcritical case) with steep potential wells were considered and some 
existence results of the solitary wave solutions were established. 

Under the conditions {Di)-{D 3 ), the System {'Px,p) has a variational struc¬ 
ture. Indeed, let 

Ea := {u G II^’^(R^) I / a{x)u'^dx < -foo}; 

Eb := {u G D^’^(R^) I j b{x)u^dx < -foo}. 

Then by the condition (Di), for every ag, bg € R and A > max{0, Ea 

and Eb are the Hilbert spaces equipped with the following inner products 

{u,v)a,x'■= / VuVu-f (Aa(a:)-I-ao)~''u'(;da;, 

Jr* 
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VmVz; + {Xb{x) + bQ)~^uvdx, 


{u,v)b.\ 



respectively, where (•)“'' := max{-, 0}. The corresponding norms are respectively 
given by 

IImIU.a := ( [ |Vup + (Aa(a;)+ao)+u^dx 

\Jr^ 



and 

Ikllh.A := f + {Xb{x)+bo)+v'^dx 

We denote the Hilbert spaces {Ea, || • IU.a) and {Eb, || • ||h,A) by Ea,\ and Eb.x 
respectively. Let E\ := Ea^x x Eb^x be the Hilbert space with the inner product 



{{u,v),{w,a))x ■■= {u,w)a,x + {v,cr)b,x- 

The corresponding norm is given by ||('u,'u)||a := (HmHq a + Ikllb a)^- Then by 
the conditions {Di)-{D 2 ) and the Holder and Sobolev inequalities, for every 
A > max{0, there exists ^a > 0 such that 


k |l2(r4) < dA |m U.A, 

k |l2(R4) < dA |f Ib.A 

(1.1) 

||m||l4(r4) < S'“5||u||a,A, 

lkllL^(RH ^ 'S'”^||w||h,A, 

(1.2) 


for {u, v) G Ex, where || • ||lp(r4 ) is the usual norm in LP{R^) for all p > 1 and S 
is the best Sobolev embedding constant from to and given by 

S := inf{||Vu||i2(R4) I u G ^^’^(R^), ||M||i4(R4) = 1}. 

It follows that Ex is embedded continuously into iJ^(R^) x iJ^(R^) for A > 
max{0, Moreover, by (ll.ll) - (ll.2l) . the conditions {Di)-{D 2 ) and the 

Holder inequality, the energy functional Jx,p{u,v) given by 


Jx,p{u,v) 

■ =\ ( iVup + (Aa(x) + i f \V v\'^ + {Xb{x) + bo)v'^dx 

2 7r4 2 Jr4 

P f 


i*dx — 


v'^dx — 


dx 


(1.3) 


is well defined in Ex for A > max{0, and /3 G R. Furthermore, by a 

standard argument, we can also show that Jx,/ 3 {u, v) is of (7^ in Ex and it is the 
corresponding energy functional to System {'Px,p)- For the sake of convenience, 
we re-write the energy functional Jx,p{u,v) by 

Ta./ 3 (w, v ) = ^Vxiu, v) - ^£/ 3 (w, u). 
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where Vx{u, v) := 'Da^\{u, u) + Vb^x^v, v) with 


'Da,x{u,v) 


(VuVw + (Aa(x) + ao)uv)dx, 


'Db,x(u,v) 


(VuVv + (Ab(x) + bo)uv)dx 


and 

£/3(m, w) := ||m||+ 2/3||m v ||li(r4). 

We are interested in finding the ground state solutions of ij’x^p) for A sufficiently 
large. 


Definition 1.2 We say that {u,v) is a ground state solution of {'Px,g) if (uyv) 
is a non-trivial solution of {Vx,p) eind the energy of {u,v) given by (11.31) is the 
least one among all that of the non-trivial solutions to (Vx,g)- 

To the best of our knowledge, the existence of semi-trivial solution to {'Px,p) 
is also unknown in the literature. Therefore, we are also concerned with finding 
the general ground state solutions to ifPx,p) for A sufficiently large. 

Definition 1.3 We say {u,v) G x is a semi-trivial solution of 

{'Px,p) if (u, v) is a non-zero solution to (fPx,p) of the type (u, 0) or (0, v); we call 
{u,v) a general ground state solution of {Vx,p) if {u,v) is a non-zero solution of 
{Vx,p) o,nd its energy is the least one among all that of the non-zero solutions 
to (Vx,p)- 

Definition 1.4 Let p,a,i and fib,! denote the first eigenvalues o/(—A, Uq (fla)) 
and (—A, iJg (rif,)), respectively. 

We denote the sets of all eigenvalues o/(—A, i?g (ila)) and (—A,i?g(nb)) by 
tT(—A, ifg (ilo)) and (t(—A, i/g (fib)), respectively. 


Remark 1.1 Without loss of generality, we always assume oq < &o throughout 
this paper. 
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1.1 The case of —/ia,i < o-o and —Hh. ,1 < ho. 

Clearly, J\^p{u, v) is heavily rely on the properties of 'D\{u^ v), uq and Bq. Firstly, 
we note that there exists Aq > 0 such that 'D\{u,v) is positively definite on Ex 
for A > Aq provided that < Qo and < bo (see Lemma 1^751 below for 

more details). In particular, Aq = 0 if oq > 0 and bo > 0. Let 

A(x./3 ■= \^iu,v) G Ex \ u ^ 0,v ^ 0, {D[Jx,i3{u,v)], {u,0))ei,e>^ 

= {D[Jx,i3{u, ?;)], (0, v))ei,e>, = o| 

and 

Mx,i3 ■■= |(w,'y) e £’a\{(0,0)} I {D[Jx,piu,v)], {u,v))e‘,Ex = o}, 

where D[Jx,i 3 {u,v)] is the Frechet derivative of the functional Ja _/3 in Ex at 
{u,v) and E’^ is the dual space of Ex- It is easy to see that ■Afx,p and Mx,p 
are both nonempty and contains all non-trivial solutions and non-zero solutions 
of the System {'Px,p), respectively. Such sets are the so-called Nehari type sets 
to {'Px,p) and they are extensively used for finding the ground state solution to 
nonlinear elliptic systems (cf. [THl HU HHl [HI HU HU HU 115] ) ■ Define 

mx ,/3 ■■= inf J\,/ 3 {u,v), a := inf J\,i 3 {u,v). (1.4) 

(u,v)£Afx^j3 ’ {u,v)£M\,13 


Since 'Dx{u,v) is positively definite on Ex, it is also easy to show that mx^p and 
p are both nonnegative for all A > Aq and /3 G R. 

Theorem 1.1 Assume {Di)-{D 3 ) and < ao,—p.b,i < bo- If ^ > Aq, then 

we have the following conclusions: 

( 1 ) . 1/0 < ao < bo, then 


mx,p 


2 (1-I-max{/3, 0}) ’ 


2(1 -I- max{l,/3}) 


for all ,5 G M. 


Moreover, both mx^p and m*x p can not he attained. 

(2) If ao < 0, then m*x p can he attained by a general ground state solution of 
(fPx,p) for all /3 G K. Moreover, there exists Ap > 0 such that the general 
ground state solution of {Vx,p) must be semi-trivial provided that one of 
the following conditions holds: 

• 0,0 < 0 < bo, (3 < 1 — and A > Ap; 

• ao < bo < 0, f3 < l3o and X > Ap, where 


j3o ■= niin 



_ iM) 

ha,l Mfc.i 


1 - 


JM 

Ub,l 


1 - 


kol 

Ua,,l 


1 - 


kol 

Ua.l 


1 - 


JM 
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(3) toa ,/3 can he attained hy a ground state solution of (’Pa,/?) if one of the 
following additional conditions holds: 

• 0-0 < bo < 0 and /3 < 0; 

• ao < 0 and jd > j3\ for some 0 < j3\ < +oo. 

Moreover, if Oq < 0, then rn\^p = m\ ^ for jd > jdx- 

The next is a by-product of the previous theorem. 

Corollary 1.1 Assume {Di)-{D 3 ) and < ao < Q,—lJ-b,i < bg < 0. If 

A > Aq, then the following equation 

— + {\a{x) + ao)u = (1.5) 

— Av + {Xb{x) + bo)v = , u £ i/^(R^), (1.6) 

have ground state solutions, respectively. 

Remark 1.2 The Corollarv \l.l\ can be viewed as the generalization of the cel¬ 
ebrated results in m obtained by Brezis and Nirenberg, where the equation is 
defined on the bounded smooth domain. On the other hand, let us recall the 
following equation which was studied in m by Benci and Cerami: 

- Au + V{x)u = u^^+^'^/^^-'^\ uGH\R^), (1.7) 

where N > ?> and V{x) is a nonnegative function. It was observed when 
V{x) = constant ^ 0, then has only trivial solution u = 0. Moreover, 

*/ll^(a;)|| i^N /2 is sufficiently small, then O) has at least one solution. 


1.2 The case of ao < —Ba,i or ho < —Bb,i 

If either ag < —Ita.i or bg < —pb,i, then there exists Ai > 0 such that T>x(u,v) 
is indefinite on Ex and has finite augment Morse index for A > Ai (also see 
Lemmabelow for more details). In this case, Afx,p and Mx ,/3 are not the 
good choice for finding the ground state solution and the general ground state 
solution of (Pa,/?)- For A > Ai, let E^x Ej^x de the negative part of 
T>a,x{u,u) on Ea,x and T>i,^xiv,v) on ifb,A, respectively. Then we can modify 
Ad A,/? to the following set 

Sa,/? := !(■«,?;)£ Pa I (-D[d^A,/?('«,■a)], (-a,u))Bj,£;,, = 0, 

{D[Jx,f!{u,v)],{w,a))Ei,Ex =0,V(u;,cr) G E^^x x(1-8) 

where Ex := Pa\(Pc^a x This kind of set is the so-called Nehari-Pankov 

type set to (Pa,/?), which was introduced by Pankov in [33] for the scalar 
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Schrodinger equations with indefinite potentials and was further studied by 
Szulkin and Weth [^. For other papers devoted to the indefinite problems, we 
would like to refer the readers to [ziiHiiia and the references therein. Define 

ca,/3 := inf Ja,/ 3 - (1-9) 

Q\,& 

Evidently, ca ,/3 > 0 whenever P > —1 since Lp{u,v) is positively definite in this 
case. 

Theorem 1.2 Assume {Di)-{D'i). Suppose either ag < with —ag ^ 

(t(—A, iJp (Da)) or bg < with —bg ^ ( 7 (— A, ilg (Df,)). If \ > Ai, then 

can be attained by a general ground state solution of {V\^p) for 0 < /3 < 1. 
Furthermore, if ag < —pLa.i < 0 < bg, then there exists > Ai such that the 
general ground state solution of [Vx^p) must he semi-trivial and be of the type 
(ux,p,0) for all X > A^. In particular, where ux,p is the ground state solution 
to the equation 


— Au{\a{x)ag)u = (1-10) 


Remark 1.3 

(a) Theorem \1.‘A only gives the existence of the general ground state solution 
to (Vx,p) for 0 < /3 < 1 and A sufficiently large in the case of ag < —pia,i 
with —ag ^ a{—A,Il}^{rLa)) or bg < with —bg ^ (t(—A, i/g(D{,)). 

However, it is still open for us that whether {Vx,p) has the general ground 
state solution for other /3 in such cases. Indeed, since Cp{u,v) is not 
symmetric in Ex due to the conditions {Di)~{D 3 ) and even indefinite on 
X for /3 < —1, Lemmas \3.6\ and \ 3. 7| which are crucial 

in the proof of Theorem \1.2\ are invalid for /3 £ (—oo,0) U [l,+oo) in 
the case of ag < —p.a,i with —ag ^ cr(—A, iJg (Da)) or bg < —fib.i with 

-bg ^ a{-A,Hl{Llb)). 

(5) By Theorem \1.2[ it is easy to show that (fPx,g) has a ground state solution 
in the case of ag < —pia,i with —ag ^ (t(—A, Ffg (Da)) and bg < —/J.b,i with 
—bg ^ (t(— A, ilg (Dfe)). However, since the dimension of the set for the 
semi-trivial solutions to (fPx,p) might be infinite, we do not know how to 
modify the Nehari type set Mx,p to some Nehari-Pankov type sets as Qx,p ■ 
Therefore, it is also open to us that whether {Vx,p) has a ground state 
solution for P ^ 0 in the case of ag < —p-a,i with —ag ^ fT(—A, Ffg (Da)) 
and bg < —p,b,i with —bg ^ (t(— A, ilg (D{,)). 

(c) To the best of our knowledge, it seems that Theorems \1.S\ is the first ex¬ 
istence result for (I1.10|) in the indefinite case. By checking the proof of 
Theorem \l.l\ (more precisely, Lemma^f^, we can also see that (11.101) has 
a ground state solution in some definite case but might not have solutions 
in the case of ag > 0. 






1.3 The concentration phenomenon as A —?■ +cxd. 

Since a(x), b{x) have the potential wells, it is natural to ask whether the ground 
state solution and the general ground state solution of (Pa./s) will concentrate at 
the bottom of a(x), b(x) as A —+oo. Our results on this aspect can be stated 
as follows. 

Theorem 1.3 Let (ma,/3, ^a,/?) be the solution oflVx^js) obtained by Theorems \ 1. 1\ 
and \1.2i Then we have the following conclusions. 

(1) If (ux^p,v\^p) is a ground state solution of {Vx,y) with j3 <0 in the case of 
oq < ^0 < 0) then up to a subsequence (ma,/ 3 ,'^a^s) —>■ (mo,/ 3 i'I' o,/?) strongly 
in X as A —>■ +oo. Furthermore, (uo,/3,uo„a) is also a 

ground state solution of the system: 

{ Am — oqu + = 0 in Via, 

Am — 6o'^’+ = 0 in fib, (l-H) 

(m,m) e Hgifla) X 

(2) If (ux,p,vx,p) is a general ground state solution of {T’x,/3) the ease of 

ag < 0, then up to a subsequence {ux,p,vx,p) —>■ strongly in 

H^{R'^) X H^fR'^) as X ^ +oo. Furthermore, (mq,/?,mq,/?) is a semi-trivial 
general ground state solution of (ll.ll|) . 

Remark 1.4 By checking the proof of Theorem 11.11 we may have fix +oo 
as A —> +00 (see Lemmas \ 3.^ and \4.5\ and Proposition \4.4\ /or more details). 
Thus, the concentration behaviors described in Theorem, \1.A\ may not hold in the 
case of fl > fix. 

1.4 Phase separation 

Note that the ground state solution and the general ground state solution of 
i'Px.p) are also depending on the parameter fl, it is natural that we are concerned 
with the phenomenon of phase separation as /3 —>■ —oo. Our results on this topic 
now read as 

Theorem 1.4 Let {ux,p,vx,p) be the ground state solution of {fPx,p) obtained by 
Theorem \l.l\ with /3 < 0. Then there exists A2 > 0 sueh that fl u\ ^dx —>■ 

0 as /? —>• —00 for eaeh A > A2. Furthermore, for every fin —>■ —00, up to a 
subsequenee, we also have the following 

(1) {ux,/ 3 „,vx,p„) {ux,oo,vx,ao) strongly in H^'-fR'^) x H^fR'^) as n ^ 00 
with MA.oo ^ 0 and ma,oo ^ 0; 

(2) ma,oo is the ground state solution of the following equation 

-Am + (Xafx) + ao)u = , u€ Rq ({ma.oo > 0}) 

while ma,oo is the ground state solution to the following equation 
-Am + fXbfx) + bo)v = v^, v € IIo({vx,oo > 0}); 
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(3) both {ua.oo > 0} and {t^a.oo > 0} are connect domains and {ma,oo > 0} = 

R^\{fA,oo > 0}. 

Remark 1.5 For the Schrodinger system in with critical Sobolev exponent 
defined in the whole space, Theorem seems to be the first result getting the 
phase separation. 

We point out that such phenomenon for the ground state solution of the 
Gross-Pitaevskii equations was observed in [niEaiiT] on a bounded domain of 
R^ or R^; and [45l |49] on the whole space R^ or R^ . Such phenomenon for the 
ground state solution of the elliptic systems with critical Sobolev exponent on a 
bounded domain in R'^(7V > 4) was involved in [lH [161 [19] . In fact, the authors 
of [in] study the system in R^ and only get an alternative theorem which can not 
assert that the phase separation must happen. In [16] (see also [H]), the phase 
separation is observed when the dimension N of R'^ is > 6 and the system is 
defined on the bounded domains. For other kinds of elliptic systems with strong 
competition, the phenomenon of phase separations has also been well studied; 
we refer the readers to [ni[ni[i3] and references therein. 

1.5 Concentration behaviors as A —)■ +cxo and /3 —)■ —oo 

We also study the concentration behaviors of the ground state solution obtained 
by Theorem II.II as A —>■ +oo and /3 —>• —oo. 

Theorem 1.5 Let (rtA,/ 3 , i’a„s) be the ground state solution of (fPx^g) obtained 
by Theorem 11.11 with /? < 0. Then for every {(A„,/3„)} satisfying A„ —>• +oo 
and Pn —i" —oo as n ^ oo, we have that (wAn./Jn!I'An./Jn) (■uo,o> strongly 
in X il^(R^) as n ^ oo up to a subseguence for some {uo^o,Vq,o) S 

Hoi^a) X Hq(LIi,). Furthermore, (mqoj'^^oo) ‘is also a ground state solution of 


The structure of the current paper is organized as follows. In section 2, 
we study the functionals 'Dx{u,v) and Cg{u,v). In section 3, we explore the 
“manifolds” Nx,p, ■Mx ,/3 and Gx.g- The section 4 will be devoted to the existence 
results. The last section is about the concentration behaviors. Throughout 
this paper, C and C will be indiscriminately used to denote generic positive 
constants and o„(I) will denote the quantities tending to zero as n —>■ oo. 


2 The functionals T>x{u,v) and jCp{u,v) 

In this section, we give some properties of Ra(m, v) and Cg{u, v). We begin with 
the study of Cp{u, v). Clearly, Cp{u, v) is positively definite on iJ^(R^) xil^(R^) 
if /3 > 0. For /3 < 0, let 

Vp = {iu,v) e H\R^) X H\R'^) I ||u||i4(R4)|hlli4(R4) > 0}, 

( 2 . 1 ) 
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then V /3 ^ 0 and it is easy to see that Cp{u,v) > 0 if and only if {u,v) G V/ 3 . 
In what follows, we will make some observations on the functional 'D\{u,v) = 
'Da,\{u,u) + 'Db^\{v,v), which are inspired by [3] and m- We first study the 
functional 'Da^\{u,u). By the condition (Hi), /g 4 (Aa(a;) + ao)u'^dx > 0 for all 
uG E\ with A > 0 in the case of oq > 0. It follows that 'Da,x{u, u) is positively 
definite on E\ with A > 0 in the case of oq > 0. When ag < 0, by the condition 
(H 3 ), we have Ha C AIa, which is given by 

A\ := {a; £ I Xa{x) + oq < 0}. (2 .2) 

Thus, Ax^% for every A > 0. Let 

Aq,o := inf{A > 0 | |AIa| < + 00 }. (2.3) 

By conditions (Hi)-(H 2 ), we can see that 0 < For A > Aq^qj we 

define 

Aa,X ■■= {u G Ea,X I suppu C 

Then by the conditions (Hi)“(H 2 ), Ea,x is nonempty and Ea,x Ea,x- Hence, 
Ea,x = ^a,x®J^aX ^-^d ^ 0 for A > Aq_o in the case of uq < 0, where is 

the orthogonal complement of Jo ,a in Ea,x- Now, consider the operator (—A + 
(Aa(x) + ao)~^)~^(Aa(x) + ao)~, where (Aa(a;) + ao)~ = max{—(Aa(x) + ag), 0}. 
Clearly, (—A + (Aa(a;) + ao)''')“^(Aa(a;) + ao)“ is linear and self-conjugate on 
^ix for A > Aa,o in the case of oq <0. By the definition of Aa,o, we can easily 
show that (—A -|- (Xa(x) + ag)~^)~^(Aa(x) + ag)~ is also compact on E^;^ for 
A > Aa,o in the case of ag < 0. Thus, by [HI Theorems 4.45 and 4.46]', the 
eigenvalue problem 

— Am -|- {Aa{x) + ag)~^u = a{Aa{x) + ag)~u on E;^x (2-4) 

has a sequence of positive eigenvalues {Q;aj (A)} satisfying 

0 < Q;a,i(A) < aa, 2 (A) < • • • < aa,j{A) -boo, as j -boo. 


Furthermore, {Q:a,i(A)} can be characterized by 


(A) 


inf sup 

dimM>i,MC.Fr^ ueM\{0} 


/jgidVup -b (Aa(a;) -b ao)+u^)da: 
/jj 4 (Aa(x) -b ag)~u^dx 


(2.5) 


for all j £ N and the corresponding eigenfunctions {ea,i(A)} can be chosen so 
that /jg 3 (Aa(x) -b ag)~e1j{A)dx = 1 for all j £ N and are a basis of J^a- 


Lemma 2.1 Assume {Di)-{Dg) and ag < 0. Then aaj(A) are nondecreasing 
in (Aa,o,-boo) for all j £ N and limA->.+oo aa,j(A) = cflj, where are the 
eigenvalues of the following equation 

— AM = Q;|ao|M, uGHl{na). (2.6) 

In particular, a® ^ *5 the first eigenvalue of (IrHI) . 
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Proof. Let Ai > A 2 > A^.o, then by the definition of Ea,\, we have i?o,Ai = 
Ea ,\2 the sense of sets. It follows from the condition (Hi) that Ea ,\2 C 
which implies C Thanks to the condition (Hi) and oq < 0 once 

more, we have 

/jj 4 (|VmP + (Aia(a;) + ao)+M^)da; ^ /r 4 (|VmP + (A 2 a(a;) + ao)+M^)da: 
J^i{Xia{x) + ao)~u^dx ~ J^i{X2a{x) + ao)~u'^dx 

for all u € Thus, by the definitions of Q;a,j(Ai) and aa,j{X 2 ), we can see 

that aa,j(X 2 ) < aaj{Xi), that is, aa,j{X) are nondecreasing in (Ao_ 0 j+oo) for 
all j € N. In what follows, we will show that limA->.+oo aaj (A) = ciaj^ where 
is an eigenvalue of (12.61) . Indeed, by the condition (H 3 ), for every j G N, 
there exists {ipm}i<m<j C such that supp(^mn supp(/j„ = 0 for m n. 

Let Mq =span{i, 5 i, • • • , ipj}. Then Mq C E;^^ for A > Aa_o due to oq < 0 and 
the condition (H 3 ) once more. It follows from (12.5|) that aa,j{X) < a* j, where 

a* J := sup J \Vu\dx \ u € Mg and J laolu^dx “ 

Since aa,j{X) are positive and nondecreasing in (Aa_ 0 :+ 00 ) for all j S N, we 
have 

lim aa , (A) = a° ■ with some a° > 0 for all j G N. 

A-n -00 ’ 

Meanwhile, by the choice of {eaj(A)}, we have 

[ {\'^ea,j{X)\'^ + iXa{x) + ao)+[eaj{X)]'^)dx < alj, (2.7) 

JR* 

which then implies that {eaj(A)} is bounded in H^’^(]R^) for A > Aa,o- There¬ 
fore, up to a subsequence, CajiX) Caj weakly in H^’^(R^) and CajiX) —>■ Caj 
a.e. in R.^ as A —>■ -l-oo. By the Fatou lemma and the condition (Hi), we have 
Jju 4 a(a;)e^ jdx = 0. This together with the condition (H 3 ), implies Caj = 0 
outside Xla and Caj G Eg(fla)- It follows from the condition (H 2 ), the Sobolev 
embedding theorem and (IQ) once more that, up to a subsequence, ea,j(X) —>■ 
Caj strongly in L^(R^) as A —>■ -boo. Now, by the condition (H 3 ), for every 
V' G C^(i2a) C E^^, we can see from a variant of the Lebesgue dominated 
convergence theorem (cf. [33l Theorem 2.2]) that 


/ Veajyipdx = lim / V ea,j{X)V'ipdx 
JUa ’ A^ + 00 Jr 4 

= lim aa,j{X) / (Aa(a:)-b ao)“eaj(A)^da; 

A^ + 00 7 r 4 

= «a J / \ao\ea,j'^dx. 

J^a 

Hence, (ea,j, a]] j) satisfies (12.61) and ^ are the eigenvalues of (12.6|) . Note that 

<aa,i(A) = inf < / (|Vup-b(Aa(a;)-bao)^M^)dx | / (Aa(x)-bao)“M^da; = 1 >. 
L Jr* Jr* 
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By the definition of the first eigenvalue to (1^ and the condition {D 3 ), we can 
easily see that q;° ^ is the first eigenvalue to (12.61) . | 

Let {aa,j} be the eigenvalues of (12.611 and {e* j} be the corresponding 
eigenfunctions. Then it is well known that aaj = where is the 

eigenvalues of the operator —A in iLg(no). Furthermore, for every og < 0, 
ka =dim(span{e* j \ aaj < 1}) is finite. Let 

^a,\ = span{ea,j(A) | aaj(A) < 1} and = span{eaj(A) | aajiX) > 1}. 

Then dim(J'_^;^) < +00 and Ea,\ = > ^a.o in tbe case 

of oq < 0. Furthermore, by Lemma [2Tl = 0 for A > Aa^g sufficiently large, 

say A > Aq > Aq^q, in the case of —^a.i < oq < 0 and ^ 0 for all A > Aq^q 
in the case of oq < where ^a,i is the first eigenvalue of (—A, iLg (fla)). 

Lemma 2.2 Let the conditions {Di)-{D 3 ) hold and oq < 0. Then there exists 
A* > Aq such that dim{iF^^) is independent o/A > A* and dim{iF^^) < ka for 
all A > A*. 


Proof. In the proof of Lemma 12.11 we obtain that Caj (A) ^ eaj weakly 
in and ea,j{X) —t ea,j S iLg(na) strongly in L^(R'^) as A —>■ +00 

up to a subsequence and limA->.+oo Q!aj(A) = Oaj-, where (ea.j(A), aaj(A)) and 
satisfy and (1^ . respectively. Since (Aa(x) + ag) < |ag| due 
to the condition (lAi), by a variant of the Lebesgue dominated convergence 
theorem (cf. [33l Theorem 2.2]), we have 

lim aa,j{X) (Aa(x) + ag)“[eaj(A)]^(ia: = a° , / jagje^ dx. 

A^. + oo 7 r4 

This together with the Fatou’s lemma and the conditions {Di)-{D 3 ), implies 
ea,j{a) —>■ Caj strongly in as A —+00 up to a subsequence. Now, 

suppose there exist j J such that 0°^ = ^ = aa^k for some /c S N. Then 

one of the following two cases must happen: 

( 1 ) — ^a,i^ 

(2) ^ ^a,2 but J'q ^ea^j^Ca^idx — 0. 
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If case (1) happen, then by uq < 0, Lemma I^TT] and the definition of eaj(X) and 
ea,i(A), we have 

= lim (aa,i(A) + aa,i(A)) 

A —>•+00 

= lira ( [ (|Veaj(A)p + (Aa(a:) + ao)+[eaj(A)]^)dx 

A^-+oo \Js^i 

+ f (|Vea,i(A)p + (Aa(a;) + ao)+[ea,j(A)]^)da; 

Jr* 

= lim / (|V(eaj(A) - ea.j(A))p + (Aa(a:) + ao)+[eaj(A) - ea.i(A)]^)dx 

A-S- + 00 

= 0 , 

which is impossible. Therefore, we must have the case (2). Let 

Ja,o = inf{j e N I a° j > 1}. (2.8) 

Then by Lemma I^TTl there exists A* > such that dim(A'^;^) = jao~ ^ is 
independent of A > A* and less than or equal to ka for A > A*. | 

Remark 2.1 Clearly, the functional 'Db,\{v,v) is also positive definite on Eb,x 
for A > 0 m the case of ho > 0. In the case of ho < 0, we can similarly 
define B\, Abp, lFb,x, QibjiX), and Then dim{F^x) < +00 

and Eb,x = Fb,x © F^x ® ^tx ^i-ll A > Ab_o- Furthermore, by a similar 
argument as used in Lemma \2.1[ we have Fjj-x = 0 for A > A^^o sufficiently 

large, say A > Af, > Ab^o? the case of —pb,i < bo < 0 and Fjj~x 0 for 
all A > Ab,o in the case of bo < —l^bp, where p.b,i is the first eigenvalue of 
(—A,iLg(fib)). By a similar argument as used in Lemma \2.‘A we also can see 

that there exists AJ > Ab such that dim{F^x} independent 0/ A > A^ and 

dim{F^x) = Jb.o “ 1 < ^6 for all X> Af 

Now, we have the following decomposition of Ex'- 

(1) Ex = Fa,x X Fb^x for A > 0 in the case of 60 > oq > 0. 

(2) Ex = {Ff x ® -^o.a) X Fb,x for A > A^ in the case of -p,a,i < ao < 0 < bo- 

(3) Ex = {Ff x © -^q.a) X {Fjj^x ® ^b,x) for A > max{Aa, Ab} in the case of 
—p,a,i < oo < 0 and —p-bp < &o < 0. 

(4) Ex = {Ff x ® ^a,x ® ^o,,x) X Fb,x for A > Aa,o in the case of ao < -pap < 
0 < 60. 
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(5) Ex = {E^x ® ® -^^.a) X (Ej^x ® ^b,x) for A > max{Aa,o, Ab} in the 

case of oo < —Mo,i and —fib,! < bo < 0. 

(6) Ex = (Ef^x ® ^a,\) X {Ej^^ © Efj^^ © Eb,x) for A > max{Ab,o, Aa} in the 
case of —fia,i < do ^ bo < 

(7) Ex = (E^x ® ® ^a,\) X (^b^A ® ® ^b,\) for A > niax{Ao_ 0 , Ab.o} 

in the case of oq < —fia,i, bo < —fib,i- 

Moreover, we have the following estimates. 

Lemma 2.3 Let the conditions {Di)-{Do) hold and ao,bo £ M. Then 

(i) 'Da,xiu, u) = ll'ull^^;^ on Ea,x and 'Db,x{v, v) = ||z;||b,A an Eb,x for all A > 0. 
(m) Va,x{u, u) > (1 - ^^)lklllA on Elx and 

Vb,x(v,v) > (1-^TT)lkllb.A 

0^b,ji,,x (A) 

on Ej^^ for all A > max{Aa, 0 j Ab,o}, where ja,x =dim{E^^ + 1 and 
jb,x =dim{E^^) + 1 . 

{Hi) 'Da,x{u,u) < 0 on E^y^ and'Db,x{v,v) < 0 on Ejj-^ for X > max{Aa_0i Ab,o}- 

Proof. The conclusions follow immediately from the definitions of E;j^^, 

Ea,x and .Fb^A, E^^x^ Eb,x- I 

By Lemma 12.31 we can see that the functional Ex{u,v) is positively definite 
on Ex in the cases of (l)-(3) and indefinite on Ex in the cases of (4)-(7). For 
the sake of convenience, we always denote 

-®A = {E^^x ® -^o.a) X (.Fb^A ® d^b,x) 

and ^ ^ 

^A = (.F,i‘;a ® E;^x ® J^a,x) X (.Fb-'^A © Ej^x ® d^b,x) 
in the definite case and the indefinite case, respectively. 


3 The sets Mx,/], Mx,p and ^a,/? 

In this section, we will drive some properties of the sets A/a,/3, AIa,/3 and Gx,/ 3 - 
We start by the observations on A4x,/3- It is well known that A4x,p is closely 
linked to the so-called fibering maps of Jx,p{u,v), which are the functions de¬ 
fined on K+ and given by Tx,i 3 ,u,v{t) = Jx,p{tu, tv) for each (u, v) G Fa\{(0, 0)}. 
Clearly, Tx,/ 3 ,u,v{t) £ C'^(M+). Moreover, T'x^p^u,v{^) = 0 is equivalent to 
{tu,tv) £ M.x,p- In particular, Ta,/3,ii,i;( 1) = 0 if and only if (m,u) £ Aix,p- 
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Lemma 3.1 Assume (Di)-{D 3 ) hold and 'D\{u,v) is positively definite on E\. 
Then for every {u,v) € i5A\{(0,0)} with £,p{u,v) > 0, there exists a unique 

tu,v — ^ {u v) ^ such that {tu^vUfiu,vV^ G and 

T\^p^u,v{tu,v) = maxt>0 TA,/3,«,w(i). Furthermore, for every {u,v) G £'a\{(0, 0)} 
with Lp{u,v) < 0, we have Xu,v ^AA\,p = 0, where Xu,v = {(tu,tv) \ t G K.’*'}. 


Proof. The proof is very standard, so we omit it here. 

Due to Lemma [331 we can see that 

* , Vxiu,vf 

mx » = mt ^ ^ , -7- 

Ex\{{0,0)} 4:Cf3{u,v) 


(3.1) 


Lemma 3.2 Let hold and 'D\{u,v) be positively definite on E\. If 

{u,v) is the minimizer of Jx,/ 3 {u,v) on Aix,p, then we have D[J\^is{u,v)] = 0 
in Ex- 


Proof. The proof is standard. Since Jx,p{u,v) is in E\, by the method of 
Lagrange multipliers, there exists G M such that D[J\^p{u, v)\—vD\^\^p{u, u)] = 
0 in Ex, where '^x,p{u,v) = {D[J\^p{u,v)],(u,v)) ei,Ex- Multiplying this equa¬ 
tion with {u,v) and noting that {u,v) G -Mx^p, we have 


ix{D[^x,p{u, v)], {u, v))ei,Ex = 2e'Dx{u, v) = 0. 

Since 'D\{u,v) is positively definite on E\, we must have ix = 0. It follows that 
D[Jx,p{u,v)] = 0 in Ex, which completes the proof. | 


We next look at the set Afx,p- From the point of the fibering maps, Afx,p is 
closely linked to the functions defined on R+ x IR+ and given by T\^p^u,v{t, s) = 
Jx,p{tu,sv) for each {u,v) G (£'a.A\{0}) x (£'f,,A\{0}). Tx,p,u,v{,t, s) G C2(M+ x 
R+)and 


dTx,p, 


’-{t,s) = 


dTx,p,, 


’-{t,s) = 0 


dt ' ds 

fi'~P 

is equivalent to {tu, sv) G Mx,p- In particular, —(1; 1) = — 
if and only if {u, v) G ■N'x,p- 


-{1,1) =0 


Lemma 3.3 Assume {Di)-{D 3 ) hold and fi < 0. If 'Da,x{u,u) and T>h,x{v,v) 
are respectively definite on Ea^x and Eb^x, then we have the following. 

(1) If{u,v) G Vx,p, then there exists a unique (tx,p{u,v), sx,p{u,v)) G K+XK+ 
such that 

{tx,p{u,v)u,sx,p{u,v)v) €JVx,p, 

where Vx,p = ExC\Vp and Vp is given by (12.11) and tx,p{u, v) and sx,p{u, v) 
are respectively given by 


tx,p{u,v) 


ll^lli4(R4)^a.A(u, u) - /3||M^u2||il(R4)T>b,A(w, v) 


(3.2) 
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and 


Moreover, Tx^p,u,v{tx^p{u,v),s\^p{u,v)) = maxt>o,s>o T’A,/3,u,'«(i, s)- In par¬ 
ticular, we have 


Tx,p,u,v{IA) = max Tx,i3,u,v{t, s) (3.4) 

i>0,s>0 

for all {u, v) £ 

(2) If{u,v) G £’a\Va,/ 3 , then Xu,v HA/a^/j = 0, where Xu,v = {(tu,sv) | (t,s) G 

R+ X R+}. 


Proof. Since 'Da^\(u,u) and 'Di,,x{v,v) are respectively positively definite on 
Ea^x and Eb^x, the proof is similar to that of [13 Lemma 3.1] and only some 
trivial modifications are needed, so we omit the details here. | 

The relation between J\fx ,/3 and Tx,p^u,v{t, s) for /3 > 0 is quite different from 
the case of /? < 0. In the case of 0 < /? < 1, we have from the Holder inequality 
that Va,/3 = i?A\{ 0 }. However, the properties described in Lemma [3.31 may not 
hold for all {u,v) G Va,/3 = i?A\{0} except (13.4p . 

Lemma 3.4 Assume {Di)-{D 3 ) hold and ft G {0 , 1) . If'Da,x{u,u) and'Db,x{v,v) 
are positively definite on Ea^x o^nd Eb^x respectively, then ([331) holds for every 
{u,v) G Mx,p- 


Proof. Suppose (it, v) G A/a ,/3 and consider the following two-component sys¬ 
tems of algebraic equations 

f Va,x{u,u) - ||M||^4(ji4)t - /3 ||u^?;^||l1(R4)S = 0, ^ 

\ Vb,x{v,v) - ||r;||l,4(R4)S - /3||MV||ii(R4)t = 0. 

Since (u, v) G Mx,p, we can see that (13.51) has a unique solution (1,1). It follows 
that (1,1) is the unique critical point of Tx^p,u,v{t, s) in M+ x 1R+. By the fact 
that /3 G (0,1), a direct calculation and the Holder inequality, we have 


and 




X,^,u,v 


df^ 


(1,1) — —2||u|||^4(r4) < 0 




dsdt ^(1,1) - 

ds^ 

= 4(lklli4(R4)|k||i4(R4) - > 0- 


I) 

dtds 


^( 1 , 1 ) 
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Note that Va^xiu^u) and 'Db^x{v,v) are positively definite on Ea,\ and Et^x 
respectively and /3 G (0,1), by Lemma [231 (ll.2|) and a standard argument, we 
can obtain that Tx,/ 3 ^u,vit, s) > 0 for |(t, s)| sufficiently small and Tx,p^u,v{t, s) —>■ 
—oo as |(t,s)| —>• + 00 . These imply that (1,1) is the global maximum point of 
Tx,p,u,v(t,s) in M+ X IR+, i.e., (13.41) holds. | 

Remark 3.1 The relation between Mx,p and Tx^p^u,v(t, s) for P > 1 is much 
more complicated than that of P < 1 and even da does not hold for some 
{u,v) G ■A4, p in this case. 

Lemma 3.5 Suppose that {Di)-{D 3 ) hold and 'Dx{u,v) is positively definite 
on Ex- If (u^v) is the minimizer of Jx,p{u,v) on Mx,p with P < 1, then 
D[Jx,p{u,v)\ = 0 m E*^. 

Proof. The proof is also standard. We only give the proof for the case of 
~Ma,i < Oo < 0 and —pib,i < 6o < 0, since other cases are more simple and can 
be proved in a similar way due to Lemma 12.31 Since Jx,p{u,v) is in Ex, by 
the method of Lagrange multipliers, there exists iyi,V 2 G K such that 

D[Jx,p{u,v)] - = 0 in E*x, 

where 

= (-D[Ja,,3(mw)],(m,0))ej,e^,«'*;^(m,i;) = {D[Jx^f}{u,v)],{0,v))Ei,Ex- 

Multiplying this equation with {u, 0) and (0, v) respectively and noting that 
{u, v) G Pf\,p, we have 

f 2j^ilhlli4(R4) +2^2/3 ||m^^^^||li(r4) = 0, 

1 2j^2|k||l4(R4) + 2viP\\u^v‘^\\L^piii) = 0. 

It follows that either vi = V 2 = Q or ||m||^ 4 ^jj 4 j = 0. 

By the Holder inequality and Lemma 13.31 we can see that Mx,p C Va^s with 
P <1. Therefore, we must have vi = V 2 = 0, which implies D[Jx,p(u,v)] = 0 in 

El- I 


Next we consider the set Qx,p- Since Qx,p is modified from Mx,p, the firber- 
ing maps Tx,p,u,v{t) also need to be modified. For every {u,v) G £iA\{(0, 0)}, 
we define 

Gx,p,u,v{w,(y,t) : F^x X Eb^x x M+ -)> M 

by Gx,p,u,v{w, <7, t) = Jx,p{w+tu, a+tv), then G{w, a, t) is G^ in xxR+, 
where u and v are the projections of u and v on 0 Fa^x and F^^^ ® Eb,x- 
In what follows, we will borrow some ideas from [39) to observe the set Gx,p by 
Gx,p,u,v{w,(j,t). 
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Lemma 3.6 Assume (I?i)-(I? 3 ) hold and either oq < —^a,! or &o < If 

0 < /3 < 1 and A > max{Aa, A;,}, then for every (u, v) € Q\,p, «,«(«, v, 1) > 

G\,y,u,v{w^a,t) for all t G (0,+cxd) and {w,a) G ^ ^b~X’ where {u,v) is the 
projection of {u,v) in iF^x ^ ^b~x- Furthermore, the equality holds if and only 
ift = l and {w,a) = {u, v). 

Proof. We only give the proof for the case of oq < —Pa,i and bo < —fJ^bp, 
since the proofs of other cases are similar and more simple due to Lemma |2.31 
Suppose {u, v) G Gx,/ 3 , t G (0, +oo) and {w, a) G F^x ^ ■^b~x- Then we have 

Jx,p{u, v) - Jx,p{tu + w,tv + a) 

1 If 

=-(Vx{u,v) — Vxitu + w,tv + a)) — - / + v'^ + 2l3u'^v'^dx 

2 4 Jr4 

+ 7 / {tu + w)'^ + {tv + + 2l3{tu + w)'^{tv + o-)'^dx 

4 Jr4 

= i((l - t^)Va,x{u,u) - 2tVa,x{u,w) - Va,x{w,w)) 

+ i / {tu + w)'^ — u^dx 

+ i((l - t^)Vb^x{v, v) - 2Wb^x{v, (t) - Vb^xicr, a)) 

+ — f (tv + a)'^ — v'^dx 

4 7r4 

+ i / 2P{{tu + w)^{tv + a)'^ — u'^v'^)dx. 

4 Jr4 
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It follows from the definition of G\,p, {w,a) G x and Lemma 1^31 that 

J\,p{u, v) - Jx^pitu + w,tv + a) 

1 If 

= --Va x{w, w) + ^ f {tu + w)'^ — + 2(1 — t^)u^ — 4tu^wdx 

2 ’ 4 7r4 

— 7 :'^b,xi<^,cr) + ^ I {tv + a)'^ — v'^ + 2{1 — — Atv^adx 

2 ’ 4 Jr4 

+ ^ f {tu + w)^{tv + a)^ — + {2 — 2t^)u^v^dx 

2 jRi 

—/3 / tv'^uw + tu^vadx 
dB<i 

> - / {tu + w)'^ + — 2u‘^{tu + w)'^ + 2u‘^w'^dx 

+ T / {tv + a)'^ + — 2v^(tv + a)^ + 2v^dx 

+ ^ f {{tu + w)^ — u^){{tv + a)^ — v^) + v'^w^ + u^a'^dx 

= 7 / f(iu + + {{tv + erf - v'^f + 

4 7r4 V 

2P{{tu + wf — u‘^){{tv + af — v'^'^dx 


+ - / w'^ + v'^ a'^ + fdv'^w'^ + fdu^a'^ dx. 

2i ItO’A 


Since (3 £ [0,1), we have 

dx p{u,v) —Jx p{tu + w,tv + cr) > j I {\{tu + wf— u'^\ — \{tv + o'f— v'^lfdx > 0 

4 7r4 

and the equalities hold if and only if t = 1 and {w, a) = (0, 0). | 


By Lemma l3.61 we have the following important observation for Gx,p- 

Lemma 3.7 Assume {Di)-{Df hold and either ag < ~Ma,i or bg < If 

0 < /3 < 1 and X > max{Ao,A{,}, then for every {u,v) G Ex, there exists a 
unique {wl,al,tl) G x ^fx x R+ such that {ulp,v^p) = (^l + t\u, + 
tfy) G Gx,p, where u and v are the projections of u and v on 0 Ea,x ond 
Efx®Eb^x- Furthermore, we also have 

G\,p,u,v{w°x,crx,t\) = ^ m_ax Gx,p,u,v{w,cr,t). (3.6) 

Proof. We only give the proof for the case of oq < — ^a.i and bg < —p.b,i, 
since the proofs of other cases are similar and more simple due to Lemma |2.31 
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Clearly, x = i^a,x ® x {J^j^x ® for every 

{u,v) G Ex with A > max{Aa, Ab}. By the definitions of E^x Ej^x^ can 
also see that dini((J'^;^0R+M) x (J'^;^0R+i;)) < 0oo for every (it, v) G Ex with 
A > niax{Aa, Af,}. On the other hand, since {u,v) G Ex with A > max{Aa, At}, 
we must have u 0 or u ^ 0. Since 0 < /3 < 1, by Lemma [23] and (11.21) . we 
have that 


> 


> 


Jx, 0 {tu,tv) 

— (Pa,\{u, U) + Vb^xiv, V)) 


Jl 

4 

t^d*x 


2 



0 


(l|■*^llL4(B4) + II^^IIl4(r4) 0 2/3||-«^U^||^1(r4)) 

«II1a + INm)( 24 - f^5-^(NllA + NIm)) 


(3.7) 


for t > 0 sufficiently small, where 


dx = min 


1 - 


1 - 


Oa,j„,x(A) ^a,jb,x{^) 


> 0 


(3.8) 


and ja,x and jb,x are given by Lemma [231 Note that 0 < /3 < 1, then for every 
{w, a) G {E;j;-x 0 K+u) x {Ej^x ® R''"^f) with llicHo + ||o-|l^ = 1, we have from 

the Holder inequality that Cp{'w, <t) > 0, which then implies 


Jx,fi{Rw,Ra) < — -^(||ii;||^4(R4) + ||ct||^4(r 4)+2/3||u;V2||ii(R4 ))-00 

as R ^ + 00 . Since dim((J'^;^ 0 R+u) x {E^x ® R'*’^)) < + 00 , there exists 
Rx > 0 such that 

Jx,i3iRxw,Rx<y) <-I (3.9) 

for all {w,(j) G {E^x ® R'*’^) x {Ej^x ® R''’^f) with ||iu||a + ||o-||j _,^ = 1. Since 

Gx,p,u,v{w,(T,t) is of in E^x ^ ^b,x ^ noting 


dim(( 0 R+u) x {Eb^x ® K'*'!')) < + 00 , 


there exists (u;°, (t°, t°) G E^x ^ ^bx ^ ®cich that (13.61) holds. It follows that 
(ic®, tT°, t°) is a critical point of Gx,p,u,v{wi cr, t) in E^x ^ ^bx ^ Therefore, 
(w^.'Cy/s) = {wx + tx^Ex +^A^' e Gx,p- Note that {ul p.vlp) G 5a,/ 3 and 
(iy°,(T°,t°) satisfy (13.61) . by Lemma [331 ( 111 °,(t°, t°) must be unique, which 
completes the proof. | 
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Remark 3.2 By Lemma lK^ and Remark \2. 1[ we have ja,\ = ja^ + 1 < fca + 1 
for A > A* and ji,^\ =j^Q + l<kb + l for A > AJ. It follows from Lemma [2A\ 
that and R\ given by (|3.8I) and (1^ respectively are both independent of A 
sufficient large, say A > Ag > max{A*, A^}. 

In what follows, we will give some estimates of J\^p{u,v) on the sets 
M.\,g and Q\p- More precisely, we will give some estimates of m\^p, m\ ^ and 
c\,p- We begin with the estimates of the upper boundary to and 

Let Iq,„(u) and In^iv) be two functionals respectively defined on Hg^fla) and 
HQ^flb), which are given by 

Iq^{u) := I- [ |Vup + i [ u^dx, 

In^iy) ■=^ [ + bov'^dx - 7 / v'^dx. 

^ Jvib ^ Jvib 

Then it is well known that Iq,,{u) and /nfc(f) are of in iLg(flo) and iLo(nb), 
respectively. Define 

Afa ■■= {u € Ro(Da)\{0} I Iq^{u)u = 0}, 

Af,:={uGRoi(D,)\{0}|4,(^)u = 0}. 

Then it is easy to show that Afa and Afb are all nonempty. Let 

ma := inf /o(w), mb = inf Ib{v). 

AA Mb 

Then it is well known that ma = in the case of uq > 0 and ma < jS^ in 
the case of < oq < 0 while mb = in the case of 6o > 0 and mb < 

in the case of —fib,! < bo < 0 due to the condition (D3) (cf. |55]L 

Lemma 3.8 Let [Di)-{Do) hold and'D\(u,v) be positively definite in E\. Then 
ma + mb > mx^/j and minjTOa, mb} > m\ ^ for all /? G K. 

Proof. Without loss of generality, we assume ma < mb- Since Afa x A/f, C A/a ./3 
and Na X {0} C AA\^p by the condition {Do), the conclusion follows immediately 
from a similar argument as used in |491 Lemma 3.2]. | 

We next give some estimates of the lower bound of m\^p and p. Let 

fw\{u) = ]^Va,x{u,u) - ^|lM||i4(R4) and Ib,\{v) = ]^Vb,x{v,v) - i||w||i4(R4). 

Then by (ll.lll - (ll.2ll . Ia,\{u) is well defined on Ea^x and Ib,\{v) is well defined on 
Eb^x respectively for A > max{Aa, A;,}. Moreover, by a standard argument, we 
can see that Ia,\{u) and Ib,>.{v) are of in Ea^x and Eb^x, respectively. Denote 

Ma,X ={UG Ea,x\{0} I I'a,xiu)u = 0}, (3.10) 

Afb,x = {UG Eb,x\{0} I Iixiu)u = 0}. (3.11) 

Then Afa,x and Afb,x are nonempty if T>a^x{u,u) and T)b^x{v,v) are positively 
definite in Ea,x and Eb^x respectively. 
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Lemma 3.9 Assume that [Di)-{D^) hold and 'Da^x{u,u) and 'Di,^\(v,v) are 
positively definite on Ea^x and E^^x respectively, then for f < 0, we have 
mx,i3 > rna,x + mb,x and m\ p > min{ma.A, where ma,x = la.xiu) 

and rrib^x = h.xiv). 

Proof. Since Va^xiuju) and 'Db,x{v,v) are positively definite on Ea^x and E^^x 
respectively and Lemma [?31 holds. the proof of mx,p > rua^x Efab^x is similar to 
uni Lemma 3.2]. For the proof of m\ ^ > min{ma,A; '^b,x}, note that by /? < 0, 
we must have mm{'Da,x{u,u) — ||rt|||,4(R4),25 &,a('*^i'*^) ~ lkllL4(R4)} ^ 0 
{u,v) £ Mx,p- It follows that there exists t G (0,1] such that either tu G N'a,x 
or tv G Mb A, which together with Lemma l3.11 implies ^ > min{ma a, rnb a}. 

I 

When 'Dx{u,v) is positively indefinite in Ex, the situation is somewhat dif¬ 
ferent. In this case, due to Lemma 12.31 we have that either ag < —pa,i or 
bo < -fJ-b,! if A > max{Aa, A{,}. 

Lemma 3.10 Assume that (Di)-(Do) hold and that either oq < —pa,i or bg < 
—fj,bp. If 0 < f < 1 and A > Ag, then we have 


cx,p > oq > 0 , 

where ag > 0 is a constant independent of P G (—1,1) and A > Ag. 


Proof. As in the proof of Lemma 13.61 we only give the proof for the case of 
ag < —fJ-a,! and bg < —p,b,i, since the proofs of other cases are similar and more 
simple due to Lemma [2.31 Let (u,v) G Gx,p- Then u = u + u and v = v + v 
with u 7 ^ 0 or ^ 0 , where u, u, v and v are the projections of u and v on 

^aX ® ^a,x, and © Eb^x, respecitvely. By a similar argument as used 
in (IX7D . we can see that 


JxAu,v) > —( 


Hix + Mixmi-t^s-^M 


11a 


kll 


lx)) 


for all t > 0, where d^ is given by (13.8|) . 


Since m 7 ^ 0 or u 7 ^ 0, there exists 
It follows that Jx,p{u,v) > 

/ ji*\2 rt2 

Note that {u, v) G Gx,p is arbitrary, we must have cx,p > 


tx £ ( 0 , + 00 ) such that t\ 

{dffSf 


\lx)=d*xSf 


^ U J V_ CXX J, VVV.. XXtA,V\.. 4 • — 

oa > 0 for 0 < /3 < 1 and A > Ag. It remains to show that oa > <ao > 0 for 
some ag independent of 0 < /3 < 1 and A > Ag. Indeed, by Lemma [22] and 
Remark 1221 ja,A = jlp + I < fca + I for A > A* and jb,x = jt^g + 1 < + 1 for 

\> Af. Then by Lemma [2Tl we have 


d*x > min 


1 - 


a, 




, 1 - 


a. 




(AS 


> 0 . 


We close the proof by taking ag = min 
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If we also have —oq ^ cr(—A,iJQ(fla)), then by the condition {D 3 ) and 
the results of [T4j |40], /q(m) has a least energy critical point Ua in HQ{Qa) 
with the energy value 0 < Ia{ua) < \S‘^. Similarly, if —60 ^ cr(—A, 
then Ib{v) has a least energy critical point Uf, in HQ(Q,b) with the energy value 
0 < Ib(vb) < \S'^ due to the condition (D3) and the results of [TilHD] , 

Lemma 3.11 Let {Di)-{D^) hold. Further, assume that either qq < —fJ-a.i with 
—oq ^ cr(—A, Hq (ria)) or bo < —p. 6,1 with —bo ^ cr(—A, iJg (Ilf,)), then we have 
that 

-5^ > limsupcA,/3 for allQ < fd <1. 

4 A^-+oo 

Proof. We only give the proof for the case of oq < —P-a.i with —oq ^ 
cr(— A, iJg (fla)), since another case can be proved in a similar way. Since Ua is 
a nonzero critical point of Ia{u) with the energy value 0 < Ia{ua) < \S'^, we 
can see that {\\7ua\^ + aou^dx > 0. We claim that there exists Aq > Ag 

such that Ua S Ea,x\iFa\ > ^0- Indeed, by the condition (II3) once 

more, we can see that Ua € Ea,\ for A > Aa,o- If there exists {A„} with 
Xn —>■ +c» as n —?► 00 such that Ua G ^a\„ i then by Lemmas 12.11 and 12.21 and 

the definition of , we must have Ua = X]i=i d-iCa^i, where j* g is given 
by (12.81) and ea^i satisfy (12.61) with Ua^i < 1 for all i = l,2,---j*g. This im¬ 
plies (|VuaP + aoUa)dx < 0 and it is impossible. Now, by Lemma 1X71 

there exists a unique G x x R+ such that = 

{wl + tl{ua-Ua,\),cr^+txivb-Vb,\)) G G\,p for A > Aq, where Ua,\ and are 
the projections of Ua and Vb in Ea\ ^'^el Ejj~x respectively. It follows from Re¬ 
mark [T 2 ] that (u>°,(T°,t°) —>■ {wo,<To,tQ) strongly in H^(W^) x x R"*" 

as A —?► -bcxD. Moreover, we also have that {ua,\,Vb^\) —>■ {ua,Vb), where 
Ua and Vb are the projections of Ua and Vb in span{e*^- | aaj < 1 }) and 
span{ej^ | abj < 1}) respectively. Due to the condition (D 3 ), we must have 
that {wo,(To) G HQ{ila) X HQ{Llb), which together with the condition (D 3 ) and 
Ua G iLg^(Da), implies 

limsup Ja,/ 3 (w° +txiua - Ua,\),crl) 

A —^ ~|~oo 

< limSUp/a,A(w° +tl{Ua - Ua,\)) < laih^Ua “ Ma))- (3.12) 

A—^~l“Oo 

By a similar argument as used in the proof of Lemma 13.61 (see also [321 Propo¬ 
sition 2.3]), we have that Ia{to{Ua - Ua)) < laiUa). Thus, \S‘^ > Ia{Ua) > 
lim sup ca ,/3 for 0 < /? < 1 , which completes the proof. | 

Next we prepare some estimates which are useful in the following sections. 

Lemma 3.12 Assume {Di)-{Do) and P < 0. If'D\{u,v) is positively definite in 
Ex, then there exists dx,p > 0 such that ||■a||^4(R4)||u|||4(][{4) ~> 
d \,/3 for all {u,v) G A/),,/?. 
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Proof. Due to Lemmas 12.31 and ESI the conclusion can be obtained by a 
similar argument as used in the proof of [491 Lemma 3.3] and only some trivial 
modifications needed, so we omit the details here. | 


Lemma 3.13 Assume that (Di)-(D 3 ) hold and 'D\{u,v) is positively definite 
in E\ in the case o/ oo < 0 < 6o. If 


0 </3 < 1 


1 

aa,i(A) ’ 


(3.13) 


then we have 


and 




“ aa,dA)) “ 


S'> 0 


a„,i(A)) 


5 > 0 


for all {u,v) € with Jx^p{u,v) < 


(3.14) 


(3.15) 


Proof. Since 'D\{u,v) is positively definite in E\, without loss of generality, 
we may assume that Ex = [^a,x ® ^a,x) x 0 Eh^x)- Suppose (u, v) € ■N'x,p 
with Jx,p{u,v) < then by Lemma ESI (11.21) and the Holder inequality, we 
can see that 


<5 < ||t^|||4(][{4) + /3 ||m|||4(r4P 


(3.16) 

(3.17) 


Since Jx,p{u,v) < we also have from Lemma ESI and (II.2|) that 

~ ;^)ll^lli*(R'‘) + ll'*^lli4(R"‘) ^ (3.18) 

<^a,l 

We can obtain (I3.14|) by (13.161) and (13.181) while (13.151) can be obtained by (13.171) 
and (13.181) due to (13.131) . which completes the proof. | 


Lemma 3.14 Assume that (Di)-{D 3 ) hold and that Dx^u^v) is positively defi¬ 
nite in Ex in the case of ao <bo<0. If 


1 - 


0!a,l(A) 


1 1 1 ' ^ ^ 

0 </3 < min<{-(1-^)(1-^),. 

'2 aa,i(A) a6.i(A) 1 - ^ 


(3.19) 


then we have 


(1-W)(l-W) - 2/3 

> .. . s > 0 


ai,,i(A)) c<„,i(A)) 


(3.20) 
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and 


Il4(r4) > 


a6,l\A))(^ aa,l(A)) q „ 

(l-^)-/3(l-^) 


(3.21) 


for all {u,v) G Mx,p with Jx^p^u^v) < ^ 5 '^. 


Proof. Since 'Dx{u,v) is positively definite in Ex, without loss of generality, 
we may assume that Ex = {iFa,x ® ^a,x) x {^j^x ® ^b,x)- Suppose (u, v) € Mx,p 
with Jx,p{u,v) < , then by Lemma [27^ (II.2|) and the Holder inequality, we 

can see that 


a„,i(A) ® (3.22) 

~ Ct6,l(A) — II'*^IIl4(R4) + /5||w|Il4(r4)- (3.23) 

Since Jx,j 3 {u,v) < \S'^, we also have from Lemmaand (II. 2|) that 

(1 ;^)l|w|li4(R4) + (1 ;^)lklli4(R4) < SS”. (3.24) 

We can obtain (13.201) by (13.221) and (13.241) while (13.211) can be obtained by (13.231) 
and (13.241) due to (13.191) . which completes the proof. | 


4 The existence results 

Note that we have assumed > ag, without loss of generality, one of the 
following four cases must happen: 

(i) bo > ao > 0 ; 

(ii) -Aia.i < ao < 0 < bo; 

(Hi) —^a,i < ao < 0, —fXb,i < bo < 0 and bg > ag; 

(iv) ao < -Ma,i or bg < -fib.i and ag < bg. 

Let us hrst consider the case of bg > ag > 0. In this case, T>x(u,v) = ||(M,r:)||A 
for all A > 0 due to Lemma [2T3l Let 

^/s(u,v) := i||VM||i2(R4) + ^||Vu||i2(R4) - ^£;3(u,v). (4.1) 

Then £p(u,v) is a functional on Z1^’^(R"‘) x Zl^’^(R^). Denote D^’^(R.^) x 
D1’2 (m 4) 2? and define 

M} := {(u,?;) e D\{(0,0)} | {D[£p{u,v)\, (u,v))-dw = 0} 
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and 


Mp := €'D\u^Q,v ^0, {D[£i}{u,v)], {u,0))'d-,v 

= p[fKn)],(0,z;))^,.,i, = 0}, (4.2) 

where D[£p{u^v)\ is the Frechet derivative of the functional £p in T) at {u^v) 
and V* is the dual space of V. 

Lemma 4.1 Let jS > 1. Then = 2 {i+p) ^‘^> where 

n{p='mi£p{u,v), m*p =mi£p(u^v). 


Proof. The idea of this proof comes from [T7]. Clearly, Np C Thus, it 
is easy to see that . Furthermore, by [TTI Theorem 3.1] (see also [l5l 

Theorem 1.5]), we know that 

m}* = for (3 > 1. (4.3) 


Thus, mPp < m*p = 2 {i+/}) for /3 > 1. On the other hand, by a standard 
argument, we also have 


m 


0 

/3 


inf max £g(tu,tv) 

(u,«)ei5\{(o,o)} t>o^ 

(l|v^lli^(«.) + l|v^lli^(K4))^ 

(u,v)&V\{{0.,0)} iCp{u,v) 


(4.4) 


It follows from the Holder and Sobolev inequalities that Next, we 

will show that > rn*^*. Let {(u„,v„)} C Aip be a minimizing sequence of 
£i 3 {u,v). Then it is easy to show that {(u„,Un)} is bounded in V. Without loss 
of generality, we assume (un, u„) ^ (uo, vq) weakly in V and {un,Vn) —>■ {uq, vq) 
a.e. in x as n —>■ oo. Denote Wn = Un — uq and cr„ = — vq. Then by 

the Sobolev inequality, the Brezis-Lieb lemma and [m Lemma 2.3], we have 


mp = £i3{uo,Vo) +£/3{Wn,an) +o„(l) 


(4.5) 


and 


0 = {D[£p{uo,Vq)], + {D[£p{Wn,(Tn% {Wn,crn))-D‘,v + 0„(1). (4.6) 

Case 1: (uo,vo) ^ (0,0). In this case, we can see from (|4.4p that 
(llVwoll i2(R4) + l|Vt^0|li2(R4))' 


-C/3(uo,Uo) 
It follows that 


> 4m^ — II Vu„l|^2(R4) + II Vn„||^2(R4) + o„(l). 


II VMo||^2(r4) + II VUo||2,2(R4) > ||'Wo|Il4(r4) + 2/3 1 1 UqUq 1 1 (R4) + 1 1 Uq 1 1 J^4(R4 ) , 
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which together with (14.1|) and (14.6|) . implies that 

£f 5 iuO,Vo) > 0 and {D[£fj{Wn, (in)], (Wn, crn))v* ,7) < 0„(1). (4.7) 

If ||Vw„||i 2 (R 4 ) + ||VCT„||i 2 (R 4 ) > C' + o„(l), then by (Ff)) . there exists 0 < < 

I + o„(l) such that € A4*^ for n large enough. Since /3 > 1, by 

Lemma l3.II and similar arguments as used in (14.51) . we can see that 

= (tn'^ 0 ? ^n'^o) 4“ ^/3 4” ^n(l) 

> £i3{tnUo,tnVQ)+m°p+Onil). 

It follows that —>■ 0 as n —?► oo, which is impossible due to ||Vw„||/^ 2 (r 4 ) + 

II V<7ra||L2(R4) > C' + o„(I) and {tnWn,tnVn) G for 71 large enough. Therefore, 
we must have ||V7 (;„||j;,2(r4) + ||V(t„||j;,2(r 4) —>■ 0 as n —>■ oo up to a subsequence. 
It follows from the Sobolev inequality and (I4.5D - (I4.7D that £i 3 {uo,vo) = and 
(uo,vo) € M*p. If ito = 0 or uo = 0, then by the Sobolev inequality and (14.41) . 
we can see that > \S^. It contradicts to < m*^ and , since (3 > 1. 
Hence, both uq ^ 0 and vq ^ 0. Since £ 0 {u,v) is C^, by a similar argument 
as used in the proof of Lemma [3.21 we have D\£jj{uQ,V[j)] = 0 in V*. Hence, 
{uq,vo) G Afp and mPp > m*p . 

Case 2: (uo, i^o) = (0, 0). 

In this case, (wn,o'n) = By ()4.5I) and we must have 

I|Vw„||l2(r 4) + ||Vcr„||i2(R4) > C + o„(l). If Wn 0 or (T„ 0 strongly in 

as n —?► oo, then we can see from the Sobolev inequality and (14.41) 
that mPp > jS"^, which is impossible since < m pA (1131) holds and /? > 1 . 
Therefore, we must have both Wn 7 ^ 0 and cr„ 74 0 strongly in as 

77 —> 00 . Now, by a similar argument as used in m Lemma 2.5], we can get a 
contradiction. | 

Due to Lemma [4.11 we can give a precise description on m\^p and niy/S 
the case of bo > qq > 0 . 

Lemma 4.2 Assume that {Di)-{Do) hold. If bg > ag > 0, then 

2(1 + max{/3,0}) 2(1 + max{l,/5}) 

for all fi G M. and A > 0. 

Proof. For the sake of clarity, the proof will be performed through the following 
five steps. 

Step 1. We prove that 777 ^ p = and mx ,/3 = for A > 0 and /? < 0. 

Indeed, thanks to the Sobolev inequality and the condition (Di), we have 
'ma,\ > \S‘^ and rub^x > \S‘^ in the case of bo > ao > 0. It follows from 
Lemmas 13.81 and 331 that 777 ^ p = and 777 a ,/3 = ^5'^ for A > 0 and /5 < 0. 
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Step 2. We prove that ^ for A > 0 and 0 < /3 < 1. Indeed, by 

Lemma [3.81 we can see that ^ for A > 0 and 0 < /? < 1. It remains 

to show that p > \S'^ for all A > 0 and 0 < /3 < 1. Suppose the contrary, 
we have rn\, p, < for some A' > 0 and 0 < /?' < 1. By the definition of 
rriy p,, there exists {us,vs) € satisfying J\'^pi{us,vs) < mx^p' + (5 for 

some S € (0, — mx\/ 3 ')- Since (us, vs) £ with A' > 0, by the Sobolev 

inequality, the condition (Di) and bo > qq > 0, we have 

Cp'iusyVs) < + AS < S^, ( 4 . 8 ) 


and 

>5'l|u5||i4(R4) + 511^511^4(84) <Vx'iu5,vs) = Ci3'{us,vs). (4.9) 

Combining (I4.8|) - (I4.9L we can obtain that ||u 5 ||| 4 (r 4 ) + lk 5 |li 4 (R 4 ) < S. On the 
other hand, thanks to the Holder inequality, 0 < /?' < 1 and (gH) , we can see 
that ||■u^|g 4 (R 4 ) + ||t' 5 |li 4 (R 4 ) > S, which is a contradiction. 

Step 3. We prove that mx^p = 2(i+p) for A > 0 and /3 G (0,1). 

Indeed, consider the following family of functions: 




2^2^ 
e2 + |a;|2’ 


e > 0. 


Then -ipeix) = 'ip*{x)r](x) G where ry G C^{Br). Furthermore, it 

is well known that ||'0 £|Il4(r 4) = 5^ + O(e^), IIVV^e|||2(R4) = 5^ + O(e^) and 
l|V^e||^ 2 ( 84 ) = o{e) (cf. [30]). It follows from the condition (Hi) that 

IIV'6lll4(R4)Ha.A(V’e, “ /?ll (V^e)'^II(V-e, V^e) = “ /? + o(e))(4.10) 

and 


IIV'e|li4(R4)H&,A(V'6,V’e) -/3||(V’e)'‘l|ii(RO^o,A(V'6,V’e) = 5^(1 -/3 + o(e)).(4.11) 
Since 

IIV'e|lL4(R4) —/3 IK'i/'e) |ll,i(R4) = (1 —/3 )||V'e|lL4(R4), (4-12) 

by (I4.10[) - (I4.11I) . we can see that the proof of Lemma 1331 still works for e suffi¬ 
ciently small in the case of /3 ^ 1 . Thus, there exist tx,/3{ipe,i^s) and sx,/3{'ipsj V'e) 
respectively given by (|3.2|) and (13.3|) such that 


{tx,/3{ll^E,i’e)lpE,Sx,l3{,i>E,1pE)i’e) G Mx,l3, 


which then implies 


mx,p < Jx,pitx,p{'ipe,i’e)i^e,Sx,p{'>pE,i^e)i’e) 


1 

2(1+ /3) 


5^ -I- o{e). 
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It follows that rn\^p < 2 ( 1 + 13 ) remains to show that m\^p > 2 {i^+p) 

Indeed, let {{un,Vn)} C A/a ./3 be a minimizing sequence of Jx^p{u,v). Since 


2 

Cp{Un,Vn) < 4mA,/3 + o„(l) < ^ ^ + o„(I). (4-13) 

Note that the condition (Di) holds, A > 0 and bo > ag > 0, by the Sobolev 
inequality and the fact that (un,Vn) € Mx^p, we have 

'5'l|rtn||L4(R4) < T^a,x{UmUn) = ||Wn||L4(R4) + ll-I/i(R'‘) (4-14) 

and 

5'|kn||i4(R4) < Vb,x{Vn,Vr,) = || U„ ||i4(R4) +/? || Li (R4) . (4.15) 

Thanks to (I4.13I) - (I4.15I) . we can see that 

2 

ll'“"lli4(R4) + ||r'n|li4(R4) < I On{l), 

which together with (14.141) and (14.151) and the Holder inequality, implies 

||u„||i4(R4) > lkn||i4(R4) > 

Since («„,«„) G ■Afx,p and /3 > 0, we must have from ()4.14p - (l4.16|) that rnx,p > 
2 ( 1 + 3 ) + On(l)- The conclusion follows from letting n —?► oo. 

Step 4. We prove that mx+ = for A > 0. 

Indeed, for every A > 0, we consider the following two-component systems 
of algebraic equations 

f T'a.A(V'e,V’e)- IIV'elli4(R4)t- ||iA^||li(r4)S = 0, 

\ Vb^xilpe.lpe) - ||'0e|ll,4(R4)S- ||V'^l|ii(R4)t = 0, 

where V'e is given in Step 3. Since ||V'e||^4(R4) = S'^ + 0{e^), ||VV’E||i2(R4) = 
S^ + Oie"^) and Ue\\l 2 (Ki) = o(s), by the condition (Di), we can see that (j4.17p 
can be solved in 1R+ x R+ for e sufficiently small and the solutions (t^, s^) satisHes 
tg -I- Se = 1 -I- o{e). Thus, we can choose te > 0 and > 0 for e sufficiently small 
such that {x/^'4’ei x/^4’e) G Afx,i- It follows that 


mA,i < Jx.iiVi^i’e, x/^A) = + o{e). (4.18) 

Letting £ —>• 0+ in (I4.18p . we have mA,i < for all A > 0. Since mA,i > rn\ ^ 
for all A > 0, by the conclusion of Step 2, we can see that mA,i > jS'^ for all 
A > 0. 
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Step 5. We prove that m\ ^ = m\^p = 2 ( 1 + 13 ) ^'^ for A > 0 and /3 > 1. 

Indeed, since A/a ,/3 C we can see that m\ ^ < mx, 0 - Note that (14.101) - 

(j4.12p still hold for A > 0 and /3 > 1, thus, we also have < 2 ( 1 + 13 ) ^'^ 

A > 0 and /3 > 1 by similar arguments as used in Step 3. In what follows, we 
will show that > 2 {i\p) ^^ for A > 0 and /3 > 1. Indeed, for every i5 > 0, 
we can take (ustVs) € -Adv,/? such that J\p(us,vs) < m\ p + S. By a standard 
argument, there exists ts > 0 such that {tsus,tsvs) G Ado. It follows from the 
condition (Di), A > 0, bg > ag > 0 and Lemmas 13.11 and 14.11 that 

g 2 

A + "Ia,/3 > Jx,l3{us,vs) > Jx,/3{tsus,tsvs) > £/3{tsUs,tsVs) > Trip = 

The conclusion follows by letting A —>■ 0"^. | 

With Lemma 1321 in hands, we can obtain the following 

Proposition 4.1 Let {Di)-{Dg) hold. Ifbg > ag> 0, then rnx,p and m\ ^ can 
not he attained for all fi gM. and A > 0. 


Proof. For the sake of clarity, the proof will be performed through the following 
four steps. 


Step 1. We prove that toa ,/3 and ^ can not be attained for A > 0 and /3 < 0. 
Firstly, we assume that there exists (ux ,/3 , vx,/ 3 ) G Mx,p such that 


Jx,f3{ux,f3,vx,i3) = mx,/3 for A > 0 and (3 <0. 

Then by Lemma l4.2l we must have 

Vx{ux,^,vx,0) = 2S^. (4.19) 


On the other hand, since {ux,p,vx,p) G Mx,p with A > 0 and /3 < 0, by the 
Sobolev inequality, the condition (Hi) and bg > ag > 0, we can see that 

ll^“A,/ 3 ||i 2 (]i{ 4 ) > and IIVuA,/ 3 ||i 2 (R 4 ) > (4.20) 

and 

l|n'A,/ 3 ||i 4 (R 4 ) > S and ||nA,/ 3 ||i 4 (R 4 ) > S. (4-21) 


By (14.201) . (14.191) . (Hi) and recall that bg > ag > 0, we have that 
ll^'*^A,/3||i2(R4) = II V?;A,/3||i2(R4) = S 

and 


/ a{x)u\ pdx = / b[x)v\ pdx = 0 . 


Thanks to the condition (H 3 ), ma ,/3 G Hg{Qa) and vx ,/3 G Hq{AIi,) and it follows 
from (|4.2ip that 


II VUA,/3||^2(r 4) ^ ll^■a|li2(R4) 

II'WA^S |lll4(R4) ueifo(Oa)\{0} ll'a||^4(R4) 
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II VtlA,/3||i2(R4) ^ ll^'*^lli2(K4) 

ll'*^'''./3|li'l(R4) '“e-f^o(f2b)\{0} ||'*^lli4(R4) 

which contradicts to the Talenti’s results in [33], since fta and fib are both 
bounded domains. Thus, toa ,/3 can not be attained for A > 0 and /3 < 0. 

We next prove that p can not be attained for A > 0 and ,5 < 0. In fact, 
assume that there exists {ux^p,v\^p) € M.\,p such that J\,p(u\^p^vx^p) = m\ p 
for A > 0 and /? < 0. Then by Lemma |32l again, we must have 

Vxiux,p,vx,p)=S^. (4.22) 


On the other hand, since {ux, 0 ,vx,p) € Mx,i 3 with A > 0 and /3 < 0, by the 
Sobolev inequality, the condition (Z?i) and bg > ag > 0, we can see that 


IIVuA„a||i2(R4) + IIVnA,/3||i2(R4) > (4.23) 


and 


II^A,/3|li4(R4) + ||i’A„a||L4(R4) > S. (4.24) 

(|4.23|) together with (14.221) . the condition (Di) and bg > ag > 0 once more, 
implies that 

II VuA„fl|||2(R4) + II VnA,/3||i2(R4) = 

and 


R4 


a{x)u\ pdx = 


/ b{x)v\pdx = 0. 
JR4 


Thanks to the condition (D3), ux,p G Hg{Qa) and vx ,/3 G iLo(Ob) and it follows 
from (I4.24P and the condition (Dg) again that 


II Vu>A,/3||^2(r4) ^ II^^IIl2(r4) 

IIl4(-][j 4) uieir^(naun6)\{o} Il'a^lli4(]j4) 

where wx ,/3 = ux ,/3 +vx,/ 3 - It contradicts to the results in [33], since fla and 
fib are both bounded domains. Thus, m*^ p can not be attained for A > 0 and 
/3 < 0. 

Step 2. We prove that mx,p and p can not be attained for A > 0 and 
/3e(0,i). 

We first prove that mx^p can not be attained for A > 0 and /3 € (0,1). Indeed, 
suppose that there exists [ux,p,vx,p) S A^a,/? such that Jx,p{ux,p,vx,p) = wa ,/3 
for A > 0 and /3 € (0,1). Then by similar arguments as used in Step 3 of 
Lemma 1321 we can show that 


||UA,/3||i4(R4) — lkA,/3||i4(R4) — ^ ^ 


S’ 


(4.25) 


and 


ll“A./3^A„fll|LHK’‘) 


1-/3 
1 + /3 


S. 
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It follows from j3 S (0,1) that 


^)lkA,/3|ll4(R4) /3 ^II'“a,/3^A,^IIl1(R4) — ^ > 0- 

On the other hand, by Lemma 2 ^a('«a,/3j i'a,/?) = It follows from the 

condition (Di) and 6o > oo > 0 that 

A f a{x)u\ odx < —and A [ b(x)v\ ^dx < — — 

Hence, there exists 0 < A' < A such that 

4(1 + 1) >i^- A')(lkA,/3||l4(R4) a{x)ul pdx 

+P\\ul,i3vl^p\\mR^) / Kx)vl^fidx) > 0 

dR4 

and 

- A')(lkA,/3||l4(R4) b{x)vlpdx 

+^I|wa,/3^^a./3||li(RO / a{x)u\pdx)>Q, 
dR4 

which implies 

a/3 a/3 

(tA',/3(uA,/3,UA./3),SA',/3(MA,/3,UA./3)) e 1] ^ [“’^1’ 

where tx' and Sa',/ 3 (ma,/ 3 , i'a./?) are given by (13.21) and (13.31) . respec¬ 
tively. Therefore, by a similar argument as used in Lemma 13.31 we must have 
that 

(tA'.^(MA,/3, V\,0)ux^p,S\i^p{u\^p,V\^p)v\^p) G A/a',/3- 

Since /3 G (0,1), by Lemmas 13.41 and 14.21 we have from 0 < A' < A and the 
condition (Di) that 

J\',/3itx',/3{ux^p,Vx,l3)ux,l3, Sx’^p{ux^p, Vx,p)vx,p) = 

That is, {tx>,p{ux^p,vx,p)ux,p, sy^p{ux^p,vx,p)vx,p) is the minimizer of Jy,p{u, v) 
on Afx\p. By similar arguments as used in (j4.25p . we can also obtain that 

||tA',/3(uA,/3,'CA,/3)uA,/3||i4(R4) = ^ I|sA',/3 (ma,/ 3,'yA,/3)^^A,/3 ||i4(R4) = 

which together with (14.251) . implies ty,p{ux^p,vx,p) = 1 and sy^p[ux,p,vx,p) = 
1. Note that {ux,p,vx^p) are the minimizers for both Jx^p{u,v) on Mx^p and 
Jx',p{u,v) onAfy,p- By A > A' > 0, /3 G (0,1), condition (Hi) and Lemma 
we can see that 


/ a{x)u\ pdx = / b{x)vx pdx = 0 . 

7r4 ’ 7r4 
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Thanks to the condition (-D 3 ) and (ua,^, ?^a,/ 3 ) G we have ua ,/3 G Ma 

and nA ,/3 G A4- Since bg > ao > 0, we must have ||'Wa,^ 1 |^ 4 (r 4 ) > S and 
lkA,^|li 4 (R 4 ) > S, which contradicts to (j4.25l) . Since ^ for all A > 0 

and /3 < 1, we can prove that ^ can not be attained for A > 0 and /3 G (0,1) 
by a similar argument as used in Step. 1. 

Step 3. We prove that mA,i and ^ can not be attained for A > 0. 

We hrst prove that mA.i can not be attained for A > 0. Indeed, suppose that 
there exists (wau,'^A. i) G A/a,i such that TA,i(MA,i,fA,i) = toa,i- By Lemma [41^ 
and the Holder inequality, we can see that ||ma,i 11 ^ 4 ( 84 ) + ll'^A.i 11 ^ 4 ( 84 ) ^ 5'. 
On the other hand, thanks to a similar argument as used in (14.81) - (14.91) . we 
have ||ma,i||| 4 (R 4 ) + ||fA,i|li 4 (][{ 4 ) < S' due to Lemma [4.21 Thus, we must have 
||uA,i||i 4 (R 4 ) + ||nA,i|li 4 (R 4 ) = S. Since (MA,i,'yA.i) G A/),,!, by similar arguments 
as used in (|4.22I) - (I4.24I) and the Holder inequality, we can see that 

/ a{x)ul^j^dx= b{x)vlj^dx = 0 , S||nA,i||i4(R4) = ||VuA.i||i2(R4) 

7r4 

and 

S||i’A,i|Il 4(R4) = II V^^A,l|li2(R4). 

By the condition {Dg), G Hq (Ha) and ua,/? G 7Lg(Hb), which contradicts to 
the Talenti’s results of [44], since Ha and H;, are both bounded domains. Thus, 
mA,i can not be attained for A > 0. Since ^ = ^S^ for all A > 0, we can 
prove that ^ can not be attained for A > 0 by a similar argument as used in 
Step 1. 

Step 4. We prove that and m\ ^ can not be attained for A > 0 and 
/3 > 1. 

We first prove that can not be attained for A > 0 and /3 > 1. Indeed, 
suppose that there exists (ua,/ 3 , ua,/?) G Mx^p such that Jx,p{ux,p,vx,p) = mx,p- 
Without loss of generality, we may assume ux,p > 0 and vx,p > 0. Clearly, 
iux,p,vx,p) G -^A./S- By a standard argument, we can see that there exists 
t\,p > 0 such that {tx,pux,p,tx,pvx,p) G Since the condition (Di) holds 

and &o > oo > 0, by Lemmas l3 .1 1 and IT^ we have that 

2(1 -I- ^ Jx,p{ux,p,vx,p) > Jx,p{tx,pux,p,tx,pvx,p) > £p{tx,pux,p,tx,pvx,p), 

which together with Lemma l4.ll implies £p{tx,pux,p,tx,pvx,p) = rnPp- Use a 
similar argument as that in the proof of Lemma 14.11 we have 

U’[^’/3(^A,/3MA,/3,^A,/3^'A,/3)] = 0 in V*. 

Therefore, by the maximum principle, we can see that tx,pux,p > 0 and tx,pvx,p > 
0 on Due to [T7| Theorem 3.1] and ,3 > 1, we must have tx,pux,p = 

t\,pvx,p = Ux,p, where Ux,p is given in El Theorem 3.1] and satisfies 

II^A,/3||i,4('r4) = II VC/a,/3||j;,2('r4) = S . 
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It follows from Lemmaand (ua,^, wa,/?) S M\^p that 
^2 {l + P)S^ 

- = J\,/3[U\,/3,Vx^i3) = - ^ : 


2 ( 1 + / 3 ) 

which then implies p = (1 + P)- By Lemmaand (itA,/ 3 , f a,/?) S once 

more, we must have 




2(1+ /3) 


= J\,p{u\^p,Vx,l3) 


= -^\{UX,I3,VX,I3) 


5^2 n 

r,f. , + / (Mx) + ao)u\p + (A6(x) + bo)vlpdx. 

+ Pj Jk* 


It is impossible since ux,p > 0, vx,p > 0 on R^, bo > qq > 0 and the conditions 
{Di)-{D 3 ) hold. Note that ^ = mx^p for all A > 0 in the case of /3 > 1, we 
can also show that m\ ^ can not be attained for A > 0 and /3 > 1 by a similar 

argument as above, which completes the proof. | 


We next consider the case of — ^a,i < oq < 0 < ho- Due to Lemma [231 we 
always assume A > Aa in this case. Let us consider the Nehari type set Mx,p 
in what follows. Since 'Da,x{u,u) and Vb^xiv^v) are both definite on Ea,x and 
Eb^x respectively for A > in the case of —p-a,i < uq < 0 < bo, we can see 
that Lemma [3.11 holds for all i?A\{(0, 0)} for A > A^ and /3 > 0. Furthermore, 
we also have the following. 


Lemma 4.3 Assume [Dx)-{Do) and —p,a,i < ao < 0 < bo- 


5^(a„,i(A)-l)^ 


A > Aa, then m*x p > 4„iax{l,/3}[a„,i(A)]2 


> 0 . 


// /3 > 0 and 


Proof. Since —^a,i < oq < 0 < 6 o for A > A^, we have Ex = {d'aX ® ^a,\) x 
Eb.x- Thanks to Lemmas l2 .1 1 and [231 and < oq < 0, for every e > 0, we 

have from the Sobolev inequality that 


e + m*x B > 


g(a„,i(A)-l) 

4Q;a,i(A) 


(llw 




Ike 




(4.26) 


for some {ue,Ve) G A4x,p with X > Aa and /3 > 0. On the other hand, since 
{ue,Ve) G A4x,p with X > Aa and /3 > 0, we can see from the Holder and Sobolev 
inequalities that 




^e|ll,4(R'‘) — 


5(aa,i(A) - 1 ) 
Q!a,i(A)max{l,/3}‘ 


(4.27) 


Combining (14.2611 and (I4.27|l . we can obtain that mx^p > 4 Ha)]^ ^ ® 
for A > Aa and /? > 0 by letting e —>• 0+. | 


35 














Proposition 4.2 Let the conditions {Di)~{D^) hold and —^a,i < ao < 0 < fep. 
J//3 > 0 and X > Aa, then J\,p{ux,p,vx^p) = Tn\ p and D[Jx,p{ux,p,vx,p)\ = 0 
in El for some {ux,p,vx,p) G Mx,p- 

Proof. Let {{un,Vn)} C AAx,p be a minimizing sequence of Jx,p{u,v). Since 
A > Aa, by Lemma r2.31 we can see that {{un,v„)} is bounded both in Ex and 
V. Without loss of generality, we may assume that {un,Vn) (uo,vo) weakly 
both in Ex and TX and (un,Vn) —> (uo,vo) a.e. in R'* x as n —>■ oo. Clearly, 
one of the following two cases must happen: 

(i) (mo,^^o) = (0,0); 

(a) (uo,vo) ^ (0,0). 

If the case (i) happen, then by the Sobolev embedding theorem and the condition 
(E 2 ), we have that 

/ (Xa(x) + ao)~ul^dx = / (Xb(x) + bo)~vl^dx = o„(l). (4.28) 

It follows from {(m„,u„)} C Mx ,/3 and (11.21) that 

‘^(ll'*^ra|lL'‘(R4) + ll'*^ra|lL4(R4)) ^ ll'^n II a, A + 11'*^" II &, A 

— ‘Exi'^n, '^n) Oa(l) 

= Cf3(Un,Vn) + 0„(1). (4.29) 

If /3 < 1; then we can see from (14.291) that ||wn|li4(R4) + lkra|li4(R4) > S + o„(l). 
On the other hand, by (HU and Lemma l3.81 we have that 

l|wn||i4(R4) + ||u„||i4(R4) < 4min{ma,m6}S'"b 

Note that min{ ma, mb} < jS'^ in the case of —/ia,i < ap < 0 < we get a 
contradiction. Thus, we must have /3 > 1. By a similar argument as used in 
Step 5 to Lemma [4^ we can see that rnl^^ < 2 {i+(i) ^'^ ~La,i < 

Op < 0 < 6p for /3 > 1. By (14.281) and Lemma lC3l it is easy to see that there 
exist 0 < < 1 + o„(l) such that (tnUn,tnVn) G Aip. Hence, by Lemma [4T] 

and the fact that {(Mn,'Cn)} C Aix ,/3 is a minimizing sequence of Jx,p(u,v), we 
can see that 

2 ^^ — ~^i^p(tnUn,tnVn) < ^(Wn, +Ora(l)- 

It follows that —>■ 1 as n —>■ oo, ^ = 2{l\p) {(tnUn, tnVn)} C M*p is 

a minimizing sequence of £p(u, v). Due to a similar argument as used in Case 2 
of Lemma l4. 11 we can get a contradiction. Thus, we must have the case (ii). In 
this case, by (EH) and the Fatou lemma, we can see that 

'^C^{uo°Vo) ~ ^ ^^Pi'>^n,Vn) + 0 ^( 1 ) > ^Cfj{uo,Vo) + 0 „( 1 ). 
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It follows that 


{D[J\,I3{uo,vo)]t{uo,vo))ei,Ex > 0. (4.30) 

Let {wn,<Jn) = (un — uo,Vn — vq). Then by the Brezis-Lieb lemma and [m 
Lemma 2.3], the Sobolev embedding theorem and n (D 2 ), we get 

(L>[JA,/3(Wn,0-„)], + (L)[Ja,/3(mo, r^o)], (uq, Vq))E l,E), = On(l), 

which together with (I4.30L implies 

{D[Jx,piWn,Crn)],iWn,an))E*,E), <0„(1). (4-31) 

Due to (13.11) and (I4.30|) - (I4.31I) . we can use a similar argument as used in the 
case (i) to show that {wn, (Jn) —t (0,0) strongly in L^(]R^) x L‘^{R‘^) as n —>■ oo up 
to a subsequence. By (14.311) . the Sobolev embedding theorem and the condition 
(D 2 ), (wmCTn) —>■ (0,0) strongly in E\ as n —>■ 00 up to a sequence. Hence, 
J\,l 3 {uo, Vq) = m\ p. Thanks to Lemmawe have that D[Jx^p{uQ, uq)] = 0 in 

which completes the proof. | 

By ProDOsition l4.21 we can see that {Vx,p) has a general ground state solution 
{ux,p,vx,p) G Ex for all /3 > 0 and A > Ao. Furthermore, we have the following 

Lemma 4.4 Let (wa,Oi^^a,o) general ground state solution of {Vx,o) ob¬ 

tained by Proposition \4-‘A Then (ux,o,vx,o) is a semi-trivial solution of {Vx,o) 
and of the type {uxp,0)- Furthermore, uxp is a least energy critical point of 

la.xiu). 


Proof. Suppose vx,o ^ 0. Since (ux.o,vx,o) is a non-zero solution of {'Px,g), by 
the condition (Di), A > 0 and &o > 0, we can see from the Sobolev inequality 
that ||'yA,o|li 4 (R 4 ) > S^. Note that the condition (D 3 ) holds, it is well known 
that ||uA.o|li 4 (R 4 ) > S'^. Hence, 

J\,l3{u\,0,Vx,o) > ^(ll'«A.o|li4(R4) -I- ||t'A.o|lL4(R4)) > -S'^, 

which contradicts to Lemma [3.81 Hence, {ux,o,v\,o) is a semi-trivial solution 
of (Pa.o) and of the type (ma,o, 0). It follows from Afa^x x {0} C A4 a,o for all 
A > Aa that ux.o is also a least energy critical point of Ia,xiu), where J\fa,x is 
given by (ITTOI) . | 

By Lemma [2.11 we have lim aa i(A) = < 1 in the case of —f^a 1 < 

A—>-+cxD ’ |ao| 

oo < 0 < 69 . It follows that for 0 < /3 < 1 ~ there exists A* > max{Ao, A;,} 
such that 0 < /3 < 1-irTY for A > AT. 

Lemma 4.5 Let {ux,p,vx,p) be the general ground state solution of {fPx,p) ob¬ 
tained by Proposition \4-^ Then we have 
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(1) (ma,o,0) is a general ground state solution of {V\^p) for all /3 < 0. 

(2) For every (3 G (0,1 — there exists Ap > A* such that (u\^p,v\^p) is 

a semi-trivial solution of {V\^p) and of the type (u\^p, 0) with X > Ap. 

(3) There exists > 0 such that {u\^p,v\^p) is a non-trivial solution of{V\^p) 
for P > Px- 

Proof. (1) Clearly, (ua,o, 0) is a solution of {Vx,p) for all P < Q. It follows 
that m\ Q > ^ for all ,8 < 0. On the other hand, since ,8 < 0, for all 

(u,u) G Mx,p, there exists 0 < t < 1 such that {tu,tv) G M\p. By Lemma [All 
we have 

J\,p{u,v) > Jx,pitu,tv) > J\p{tu,tv) > m*x Q, 

which implies ^ q for all ,8 < 0. Therefore, (wa.o, 0) is a general ground 
state solution of ifP\,p) for all /3 < 0. 

(2) Suppose the contrary, there exists {A„} with A„ —>■ +oo as n —?► oo and 
P G (0,1) such that {ux„,p,vx^,p) G Afx„,p and 

•/a„./3(ua„,/3,i^a„,/3) =m*x^,p- 

Thanks to Lemma [3.81 we have that {(uA„„a, UA„„a)} is bounded in 

V = X 


Without loss of generality, we assume {u\^^p,v\^^p) {uei,p,VQ,p) weakly in T) 
and (ma„,/ 3 , ^^A„„a) — >■ (uo,/ 3 , uq,/?) a.e. in xR^ as n — >■ oo. Note that A„—>■ +oo 
as n —>■ oo, we can see from the condition [Di) and Lemma 13.81 once more that 



A„,/3 + K^)'^\„,pdx —>■ 0 


as n ^ oo. 


(4.32) 


By (T> 3 ) and the Patou’s lemma, we see that (uo^p,vo,p) G iLQ(Oa) x Hq{XIi,) 
with uq^p = 0 outside fla and vo,p = 0 outside fib- It follows from the 
Sobolev embedding theorem, the condition (D 2 ) and (|4.32l) once more that 
(ua„,/3,ua„,/3) — >■ {uo^p,vo^p) strongly in L^(R^) x L^(R"^) as n — >• 00. Since 
TLq (ria) X TLg (fife) C Ex, by the condition {D 3 ), it is easy to see that = 0 

and /((uo./s) = 0 in and H~^{nb), respectively. Thus, we have from 

Lemma 13.81 that 


imn{ma,mb} (4.33) 

> lim sup rn*x^ ^ 

n—^oc 

> liminf ^'T>x^{ux^,p,vx,„p) 

n—>-oo 4 

^ ^(ll^“o,/3|li2(R4) + ao||Mo./3||i2(R4) (4.34) 

+ 11 Vuo,/3||i2(j54) + ^o|ko./3|li2(R4)). (4.35) 
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Since fit is bounded and —/ia,i < oq < 0 < 6 o, it is well known that Ib{vQ,p) > 
mb = if fo ,/3 ^ 0, which together with (14.351) and the condition (£> 3 ) once 
more, implies that rig,/? = 0 and is a least energy critical point of Ia(u). 
Thanks to (I4.35P and (ma„„s,—>■ (wo„a, t'o./s) strongly in x 

as n — >• 00 once more, we can see that (ma„., 9; I'An,/?) (ito,,g, 0 ) strongly in 

X H^{W^) as n — >■ 00. Since 0 < /? < 1 — ^ ^ for n sufficiently large, 

we can see from Lemma 13.131 that 


lkA„./3|| 


0 ^ 


( 1 - 


l(An 




1-/3(1- 


-S = 


i(A„) 




.( 1 ), 


which is a contradiction. 

(3) By (13.ip and the condition (£> 2 ), we can see that 

{T^a,\{u\,o, U\fi) + £’f),A(MA, 0 : ^^A.o))^ 

8(l + ^)hA,olli 4 (K 4 ) 

2ma,A 00 “1“ ^00 ) + 6o-ao)||MA.o|li2(R4) 

(A(floO “1“ ^00) H“ ^0 ^0) II ^A,0 II 

^ 8(l + /3)||wA,o|li4(„4) ’ 

where ma,\ = infA/^.^ Ia,\iu). Thus, m\ p —>• 0 as /? —>■ +00. By — /ia,i < og < 0 
and Lemma [231 we have ma,\ > 0 for all A > Aa. Thus, there exists / 3 a S 
( 0 , +00) such that m\ p < ma,\ for j3 > j3\. Since < og < 0 , it is easy to 

show that ma,A < If {u\^p,v\^p) is of the type (0 ,ua,/3 ) for some /3 > f3\, 
then by a similar argument as used in the proof of Lemma 14.41 we can see that 
wa ,/3 > \S'^, which is impossible. If {u\^p,v\^p) is of the type (ma,/3 ,0) for some 
f3 > (3 then also by a similar argument as used in the proof of Lemma 14.41 
we can obtain that m'^ p > ma,\, which is also a contradiction. Therefore, 

{u\^p,v\^p) must be a non-trivial solution of (Vx^p) for j3 > j3x- I 


"iA./3 < 

< 


Next, we consider the case of < og < 0, —^ib,i < 6 g < 0 and 6 g > og. 

Due to Lemma 12.31 we always assume A > max{Aa, A{,} in this case. 

Lemma 4.6 Assume {Di)-[D^) and —/ia,i < og < 0 and —Hbp < 6 g < 0. 
Then limA-s.+oo 'ma,x = "ma and limA-s.+oo 'mb,\ = m-b, where ma and mb are 
given by Lemma \3.8\ and ma,x and mb^x are given by Lemma \3.9\ 

Proof. We only give the proof of liniA^+oo ma,x = ’m-a- Due to the condition 
(Di), it is easy to show that TOq,a is nondecreasing by A. Thus, combine with 
(D3), it implies hmA->.+oo ^ ^a- By Lemma [4^ nia.A can be attained by 
some uxfi € Ex for A > A^. Now, thanks to a similar argument as used in the 
proof of (2) to Lemma 14.51 for every A„ —>• +00 as n —>■ 00 , we can see that 
ua„.o ^ uo.o weakly in and ux„,o uo,o a.e. in and ux„,o —>• ug_g 
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strongly in as n —>■ oo up to a subsequence. It uq.o = 0, then by the 

condition {D 2 ) and (I4.32|) . we have 

‘ 5 '||uA„,o||i 4 (R 4 ) < ||uA„,o||a,A = 2 ^a,A(MA„. 0 ,'«A„,o) + 0„(1) = ||'«A„,o|li 4 (R 4 ) + 0„(1). 

It follows that ||uA„,o|li 4 (R 4 ) > + o„(l). Thus, we must have that ma,A„ > 

+ o„(l). It is impossible since rria < \S'^ due to oq < 0. Thus, we must 
have Mo,o 7 ^ 0. By a similar argument as used in the proof of (2) to Lemma 14.51 
we have that /'(ito,o) = 0 in Therefore, limA ^+00 Wa.A = Wo- I 

Thanks to Lemma [2.3l we can see that every minimizing sequence of v) 

on A/a ,/3 is bounded in E\ in this case. Using Lemma 13.121 the implicit function 
theorem and the Ekeland principle in a standard way (cf. |15j). we can obtain 
a (PS)mx,fi sequence of J\^p{u,v) in A/a ,/3 for /3 < 0, denoted by {{un,Vn)}- 

Proposition 4.3 Let the conditions (Di)-{D^) hold and /3 < 0. Then 
can he attained by a ground state solution of (Pa,/?) for X > max{Ao, A/,}. 

Proof. Since {(un,'yri)} is bounded in E\, without loss of generality, we 
may assume {un,Vn) {uq,vo) weakly in as n —>■ 00 . It follows that 
D[Jx,i3{uo,vo)] = 0 in El. 

Case 1: uq = 0 and uq = 0. In this case, by a similar argument as used 
in the proof of Proposition 14.21 we can obtain that ||Mn|li 4 (R 4 ) > S + o„(l) 
and ||i'n|||4(R4) > S' + o„(l). It follows that wa,/? > ^S^, which contradicts to 
Lemma 13.81 

Case 2: uq = 0 and vq ^ 0. Let (t„ = Vn — vq- Then (u„, (t„) ^ (0, 0) weakly 
in Ex as n —7 00 . It follows from the Sobolev inequality, the Brezis-Lieb lemma 
and m Lemma 2.3] that 

J\,p{Unj Vn) = Jx,p{Unj <7n) + Ib,\{vo) + 0„(1), (4.36) 

(P \Jx.P ^n)] t (Ttl ; 0))i?^ .E\ — (P[SA,/? {p^n 5 ^n)]: {p^n-! 0))i?^ .E\ T^n (1); (4.37) 
(P ['7 a,/3 (p^n 5 )]; ( 0 ; ^n))i?^ 

= {D[Jx,p{Un,Vn)], {S>,Vn))El,E^ + + 0„(1). (4.38) 

Since u^Vn ^ 0 in (R"^), by D[Jx^p{um Vn)] = o„(l) strongly in as n —7 oo, 
we must have ^{vo)vo = 0. It follows that h.xivo) > mt^x- If ||o'„||i4(R4) > 

C + o„(l), then by a similar argument as used in the proof of Proposition 14.21 
we can see from (14.371) and (14.381) that ||'an||^ 4 (R 4 ) > S + o„(l) and ||o'n|li 4 (R 4 ) > 

S + o„(l), which together with (14.36p . implies mx,p > ^S^. It contradicts to 
Lemma [3.81 Thus, we must have tT„ —>■ 0 strongly in L^(R^) as n —>■ oo. Since 
uq = 0, we still have ||Mn|li 4 (R 4 ) > S + o„(l). Now, by (|4.36|) once more, we can 
see that mx,p > + to;,,a. Thanks to Lemmas 13.81 and HU it is impossible 

for A sufficient large. Without loss of generality, we assume mx^p > + nib^x 

can not hold for A > max{Aa, Ab}. 
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Case 3: uq ^ 0 and wq = 0. We can exclude this case by a similar argument as 
used in the Case 2. 

Combine with the above 3 cases, now we must have uq 0 and vq ^ 0. It 
follows that (uo,vo) € A/a,/ 3 , which together with the Fatou’s lemma, implies 
that Jx,/ 3 iuo,vo) = m\^ 0 . Thus, (uo,uo) is a ground state solution of (T^a,/?) for 
A > and /3 < 0. | 


We next consider the general ground state solution of {V\,p) in the case 
of < oo < 0, —Atfc.i < foo < 0 and bg > oq. By Lemma [2.11 we have 

liniA^+oo Q!a,i(A) = ^ < 1 and limA^+oo ab,i(A) = ^ < 1 in this case. Let 


ido := min 


1 - 


|f>ol 


X _ l°ol 

Ma,l 


1 /., l^oL - Mb,i 

2 ^ 


Then it is easy to see that /3o < 1- It follows that for 0 < /3 < /Iq, there exists 
A 2 > max{Aa, A{,} such that 


0 < /I < min 



1 

aa,i(A) 


)(1 


1 

0 : 6,1 (A) 


), 


1 - 


1 

ai,,i(A) 

1 

“a,l(A) 


1 - 



(•^) 


as long as A > A 2 . By checking the proofs of Proposition 14.21 and Lemma 14.51 
we can see that they still work for A > A 2 and /? < /3o since Lemma [3.141 holds. 
Thus, we can obtain the following. 


Proposition 4.4 Let the conditions {Di)-{D 3 ) hold and /3 > 0. If < 

ao < 0, < bo < 0 and A > max{Aa, A6}, then Tn\ p can be attained by a 

general ground state solution of ifPx^p). Moreover, we have the following. 

( 1 ) The general ground state solution of (Pa,o) 0 , general ground state 

solution of (Pa,/?) for /3 < 0. 

(2) For every /3 £ (0,/3o), there exists Kp > A^ such that {ux,p,vx,p) is a 
semi-trivial solution of {Vx,p) and of the type (ua,/ 3 , 0 ) with \> Ap. 

(3) There exists /3x > 0 such that m\ p can he attained by a ground state 
solution of (Pa,/?) with /3 > Px- 


Now, we are ready to give the proof of Theorem 11.11 
Proof of Theorem ll.lt In fact, it is a straightforward consequence of Lem¬ 
mas |321 and and Propositions 14. | 

In the following part of this section, we will consider the case of ao < —Ma,i 
or bo < Let Gx,p be the Nehari-Pankov type set related to Mx,p, which 
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is given by m- Then it is easy to see that Gx,p contains all non-zero solutions 
of In what follows, we will borrow some ideas from [39] to show that 

ca ,/3 = inf^;, ^ J\^ij{u,v) can be attained by some non-zero solutions of {V\^p) 
for A sufficiently large and 0 < /3 < 1. Denote the map (m,u) —>■ (ma,/ 3 i^> 7 
(m° p{u,v),m\ p{u,v)), where (u° p) is given by Lemma IffiTl Then we can 

obtain the following 

Lemma 4.7 Assume {Di)-{D^) and oq < or bo < d/0 < /3 < 1 

and A > Aq, then the map {u,v) (m° ^(u, z;), ^(u, w)) is continuous on 

Ex, where Aq is given by Lemma \3.1(A 

Proof. We only give the proof for the case oq < —A^a,! and bo < —pb,i, 
since other cases are more simple and can b^ proved in a similar way due to 
Lemma [2T^ Let (w„, Vn) —t (w, v) strongly in E\ as n ^ oo. By Lemma iTTl we 
can see that 


{rh°xp{Un,Vn),rhlj3{Un,Vn)) = {w^+Cun,(T°+t°^Vn) 

and 

irh°x p{u, v), m°x p{u, v)) = {w° + t°u, cr° -f t°E). 

Since 0 < /3 < 1 and dim(J'^;^ 0]R+zt) x ©M+zJ) < -|-oo due to LemmaH^l 
there exists Rx > 0 such that Jx,p{Rxw, Rx(r) < —1 for all [w,(j) € [^aX ® 
R+zt) X ©M+z;) with ||w||2 ;^ + ||cr||^_^ = 1. Since {un,Vn) -)■ {u,v) strongly 
in Ex as n ^ oo, we have Jx, 0 {Rxw, Rxor) < 0 for all (zc, u) G [EaX ® ^ 

® M+zIn) with ||zzi||^ + ||z7||j A = 1 ^ad n sufficiently large. It follows from 

that {{w^ + t'^Un, cr°+t°z;„)} is bounded in Ex- Without loss of generality, 
we may assume that (z(;° + t°zz„, cr° + t°z;„) ^ (zCq + t^u, CTq + tozJ) weakly in Ex 
and (zz;° + t°zz„, cr° + t^Vn) —>■ (wq + tgU, o-g + t^v) a.e. in x as n —>• oo. 
Since dimF^_,^ x < +oo and (ztn, Vn) —>■ (zz, v) strongly in Ex as n ^ oo, we 
have {w° + t'^Un, + t^Vn) —>■ (wq + tgzz, (Tg + tgz;) strongly in Fa as n —>■ oo. 
Now, by (1^ . we can see that 

Jx,0(w° + t^Un, cr° + t^Vn) > Jx,/3{w° + t°Un, ( 7 ° + t°Vn) 

= Jx,i 3 {w° + t°u, a° + t°v) + o„(I). 

and 

Jx,p{w° + t°U, cr° + t°v) > Ja,/3(Wo + tgZZ, (Tg + tgZT) 

= JxA'^n + CA, + t°zJ„) + 0„(1). 

Note that {w°,a°,t°) is the unique one satisfying (|3.6I1 for {u,v), we must have 
zcg = w°, (Tg = cr° and tg = t°. Since {un,Vn) {u,v) strongly in Ex as 
n —>• oo, we can see that the map (rh® ^{u,v),'fh\ is continuous on Ex 

for 0 < /3 < 1 and A > Ag. | 
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Let 


1+ :={(«,«) ei?A I h|llA + IIHlM = l} 

and consider the following functional 

$A,/3(w,'c) := Jx^p{m^x^0{u,v),ml p(u,v)). 

Then by Lemmas 13.71 and HT71 $a ./3 («,!') is well defined and continuous on 
for 0 < /? < 1 and A > Aq. Furthermore, we also have the following 

Lemma 4.8 Assume that {Di)-{D 3 ) hold and that either gq < —^a,! or bo < 
If 0 < P ^ and A > Aq, then $A,/ 3 (M,f) is on Bf. Moreover, 

(A'[$a./3(u, ?;)], {w, (r))Ei,E^ = {D[Jx,fi{rh\p{u, v), m%{u, u))], {t\w, tl^))E-,E>. 

(4.39) 

for all {u,v) and {w,^) G B)^, where G ]R“*" is given by (1^ and only depends 
on {u, v); w and a are the projections of w and a on d^ax®^a-,x and iFjj-^(BiFb,x- 

Proof. We just give the proof for the case oq < —fJ-ap and bo < —fJ-b,i- Let 
{w, a) and (u, v) G Bj*". Then w = w + w, u = u + u and ct = (t + ct, v = v + v 
with u ^ 0 or F 0 and w 0 or ct ^ 0, where uJ, u and a, v are the projections 
oi w, u and ct, u on and while w, u and a, v are the projections of w, 

u and (T, V on T^x ®^a,x and J^b~x ®-I^b,x- By the implicit function theorem, it 
is easy to see that there exist <5 > 0 and a function t{l) on {—6,6) satisfying 
t{l) G [ 5 , 1 ], t(0) = 1 and t'(0) = -{{u, 'w)a,x + {v, a)b,x) such that {wi,ai) G Bj*" 
for I G {—6,6), where {wi,ai) = {t{l)u + lw,t{l)v+ la). Now, by the mean value 
theorem and Lemma EJl we can see that 

^x,p{u,v) - ^x,/3{wi,ai) 

< Jx,f!{wl + tlu,al + tlv) 

-J\,p{wl + tl{t{l)u + lw),al + tl{t{l)v + la)) (4.40) 

= -{D[Jx,i}{pi{l),P 2 {l))\,{tl{{t{l) - l)u + lw),tl{{t{l) - l)v + la)))Ei,E>, 


and 


^x,p{u,v) - ^x,p{wi,ai) 

> Jx,p{w%i + t'x.i'a, al l + txyv) 

— Jx,p{wxy + t°xy{t{l)u + lw),axy + t°_;(t(Z)^^ + la)) (4-41) 

= -i{D[jx,p{pui),pm)]Atii{W) - i)u+iw)AA{m - i)v+imEi,E>„ 


where 


pi{l) — + tx{t{l)p} + (1 — p}))^ + l{l — p])w, 

P 2 { 1 ) = A + tl{t{l)pf + (1 - pf))v + 1{1 - pAa, 
Pi(0 = wx,i + + (1 - p]'*))u + i{i- p]'*)w 


43 


and 


pW) = ol, + + (1 - pI*))v + - pI*)-^ 

with pj, pf, p^’*, p^’* G (0,1). Since (wi^ai) —>■ {u,v) as Z —>■ 0, by Lemmas 13.71 
and 14.71 and (|4.40I) - (I4.41L we have from (to° ^(u,?;),z;))) £ Q\^p that 

^A,/3(w;,g;) - $a,/3(m,Z;) 

;->o+ I 

= {D[Jx,p{rhl p{u, v), ml p{u, v))], {tl{t'{0)u + w), {0)v + a)))Ei,E^ 

= {D[Jx,piml p{u, v), rh%{u, z;))], {t\w, t°x^))Ei,E^, 

where t\ G K+ is given by (13.6p and only relies on {u,v). It follows that 
^x,p{u,v) is of on B)*' and (14.391) holds. | 

Thanks to Lemma l47^ we can obtain the following 

Proposition 4.5 Let {Di)-{D 3 ) hold. Assume that either oq < —Pa,i with 
-ao ^ a(-A,JLg(fla)) or bo < -p.b,i with -bo ^ (t(-A, ILq (14;,)). If 0 < fd < 1 
and A > Aq, then cx,p can be attained by a non-zero solution of (fPx.p)- 


Proof. We only consider the case oq < —Pa,i with — oq ^ a{—A, Ho{fla)) and 
bo ^ ~Pb,i with —bo ^ o^(—A,IIg(^}b)), the other cases are more simple in view 
of Lemma 12.31 Let 

CA,/3 = inf ^x,p(u,v). 

B+ 

Then by Lemmas 13.61 and iTTl it is easy to see that cx,p = cx,p. Since is 
due to Lemma 14.81 we can apply the implicit function theorem and the Ekeland 
principle in a standard way (cf. [15]) to show that there exists {(zx„,z;„)} C B)*' 
such that ^x,p(un,Vn) = cx,p+o„(l) and (L)[$a,/ 3 (zz„, z;„)], (zn, cr))^*,^;^ = o„(l) 
for all (w, (t) G B)*". For the sake of clarity, the remaining proof will be performed 
by several steps. 

Step 1. We prove that {m° ^(zz„, z;„), zfz° ^(zz„,z;„)} is bounded in Ex- 

For the sake of convenience, we denote (m° ^(un, Vn),rrfl p{un, n„)) by {4>n,'4’n)- 
By the definition of $A,^(zz,n), it is easy to see that {4>n,'ipn) G Q\,p and 
J\,p{4>n,tpn) = CA./3 + 0„(1). It followS that 

CA,/3 T On(I) = V^n) = f^n)- (4.42) 

If ll</>ra||L*(R^) + l|V'ra||L<‘(R'i‘) +00 as 71 —?► oo up to a subsequence, then by 

(|4.42|) . 0 < /3 < 1 and the Holder inequality, we can see that 


.(!) = 


CA,/3 


.(1) 








li'‘(R<‘) 


>-(l + /3)>0, 
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which is a contradiction. Thus, is bounded in x Since 

A > Aq > max{Aa^0i Ab_o}, by the definitions of Aa_o and At,_o given by (I2.3|) 
and Remark O we can obtain that 

/ {Xa{x)+ao)~(f>ldx <\ao\\Ax\^\\(l)n\\l4iK4), (4.43) 

f {Xb{x)+bo)-v^|;ldx <\bo\\Bx\^\\'iP,,\\%(^^4), (4.44) 

where Ax and Bx are given by (|2.2p and Remark l2.1[ This implies 

CA,/3 + On(l) 

~ ^(^a,A(0n; 0n) “t“ V^n)) 

> liUnWlx + UnWlx) - liKWAxl^UnWh^^.) + 1 ^ 0 11^. | ^ 1| || i4(R4) ). 

Hence {{4>n,'4’n)} is bounded in Ex- 

Step 2 •We prove that {cf’m'ipn} is a {PS)c^j 3 sequence of Jx,p{u,v). Indeed, by 
the definition of 4 'a./ 3('«, t'), it is easy to see that 

V^n) — ^A,/3 4” 0^(1). 

It remains to show that IA[JA,,g(^!'n, V'™)] = On(I) strongly in Let {ip, 77) € Ex- 
Without loss of generality, we may assume {ip, 77) ^ (0,0) and ||v^||q + || 77||f = 
1. If ((/7,?7) e X then due to {4>n,il^n) S Gx,p, we have 

{D[Jx,p{4>n,i’n)],{‘P,r]))E*,Ex =0 for all 77, £ N. 

Otherwise, by Lemma iTSl we can see that 

{D[Jx,l3{(l)n, i^n)], {t%n^, E*,E^ = 0 „( 1 ), 

where ip and rj are the projections of (p and 77 on E^^ © Ea,x and Ejj-^ © Eb,x 
and „ £ R'*' is given by (EH) and only depends on {(j)n,'4’n)- If n 0 
77 —>■ 00, then by Lemma [2.31 (13.61) and Step 1, we must have Jx,p{4>n,'^n) < 0, 
which implies cx,p < 0. It is impossible since Lemma [3.101 holds for A > Aq. 
Therefore, by {4>n,tpn) £ Gx,p, we have 

{D[Jx,f!{(l)n, ipn)], {ip, v))ei,e^ = o„(l) for all {ip, 77) £ Ex- 

Step 3. We prove that there exists {ux,p,vx,p) £ Gx ,0 such that 

D[Jx,piux,p,vx,p)] = 0 

in E^ and Jx,p{ux,p,vx,p) = ca ,/3-Without loss of generality, we may assume 
that (</)„,-^n) ^ {ux,p,vx,p) weakly in Ex and (</>„, V'n) -t {ux,p,vx,p) a.e. in 
R^ X as 77 —)► 00. Clearly, D[Jx,j3{ux,p,vx,p)\ = 0 in E"^- In this situation, 
two cases may occur: 
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( 1 ) {ux,p,vx,p) = ( 0 , 0 ); 

(2) {ux,i3,vx,p) ^ (0,0). 

If case (1) happens, then ((/)„,'*/'«) ^ (0,0) weakly in Ex as n ^ oo. Since 
I^IaI < +00 and \Bx\ < +oo, by the condition (Z?i), we can see that 


{\a[x) + aa)4>ndx > / (Aa(a;) + ao)“^(/)„fia; + o„(l) > o„(l) 


and 


{Xb{x) + bo)’ijj^dx > / {Xb{x) + boytjj^dx + o„(l) > o„(l). 


where Ax and Bx are given by (12.21) and Remark [2.11 By using the Sobolev 
inequality and { 4 >n, 4 ’n) £ we have that 

‘5'(ll^^n|lL4(][{4) + ||'0ri|lL4(][{4)) < \\<Pn\\ L4(]g4) + ||'0n|| L4(]g4) + 2/3|| || j;,! (r4) + 0„(1). 

Note that 0 < /? < 1 and Jx,p{4’m'^n) = ca ,/3 + o„(l), by Lemma 15.101 we have 
S + o„(l) < ||«f)n|li4(R4) + ||'0n|li4(R4), which then implies 

ll?^n|li4(R4) + ||'0n|lL4(R4) + 2/3||())^'!/'nl|LHR^) > + 0„(1). 

Now, by Lemma [3.Ill we can see that 

^*5* > Ca,/ 3 “k Oyi(1) = '^A,/5(*5^n, V^n) 

= ^(ll<(’n|lL'‘(R4) + ||V'n|lL4(R4) + 2/3||(/)^^^||il(R4)) 

^ ^‘5'^+0„(l), 

it is impossible. Therefore, we must have the case (2). In this case we can easily 
see that Jx,p{ux, 0 ,vx,p) > ca,/ 3 . On the other hand, since 0 < /3 < 1, we must 
have 

+ \i’n\'^ + 2/3|(()„|^|V'n|^ >0 all n £ N and x £ 

It follows from the Fatou’s lemma that 

liminf [ {\(l)n\^ + \tj}ri\‘^ + 2P\cj)n\'^\l{jnf)dx 

n^ca 

> [ i\ux,p\‘^ + \vx,l3\'^ + ‘2f3\ux,l3\‘^\vx,l3\‘^)dx, 

Jr* 
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hence 


^A,/3 — lini '4^n) p. ['^A,/3 (0n 5 V^n)] 5 (^n 5 V^n))Bt 

n—^oc z 

= i lim / {{(pnl'^ + \4’nf + ‘2l3\(/)nf\tlJn\'^)dx 
4 n^oo 7^4 

>7/ {.\u\,p\^+ \vx,l3\‘^ + 2f3\ux,i3\'^\vx,i3\'^)dx 
4 7r4 

= Jx,p{ux,p,Vx,p) - -j^{D[Jx,p{ux,p,Vx,p)],{'^\,0^'^>;0))El,Ex 
= Jx,p{ux,p,Vx,p)- 

Therefore, {ux,i 3 ,vx,p) is a general ground state solution of {Vx,p) for A > Aq 
and 0 < /? < 1. I 

For the general ground state solution of {Vx,p) obtained by Proposition 14.51 
we also have the following properties. 

Proposition 4.6 Let (ux,i 3 ,vx, 0 ) be the general ground state solution of {Vx,p) 
obtained by Proposition \4-5\ Then 

(1 ) (ma.OjI'a.o) *s a semi-trivial solution of {Vx,o) and must be of the type 

(ma.OjO) in the case of ao < —pia,i with —ag ^ Ffg (Ida)) and bg > 0. 

Furthermore, (ua,Oi0) a least energy critical point of Ia^x(u). 

(2) For every 0 < < 1, there exists > Ag such that {u\^p,v\^p) is a 

semi-trivial solution of {Vx,p) and must be of the type (wa,/ 3 , 0) for A > A^ 
in the case of ag < —pia.i with —ag ^ cr(—A, Ffg (0^)) and bg > 0. 

Proof. (1) Since Lemma [3.111 holds, by a similar argument as used in the 
proof of Lemma 14.41 we can get the conclusion. 

(2) Suppose the contrary, (ua„,/ 3 o, ^^a„,/3o) is non-trivial for some /3o £ [0,1) 
and {A„} with A„ —>■ +oo as n —>■ oo. Since Lemma [3 .11 1 holds and Pg £ [0,1), 
by a similar argument as used in Step 1 of the proof to Proposition 14.51 we 
can see that {{ux,,,po,ux„,Po)} is bounded in x Without loss of 

generality, we assume (ma„,/3o.'*^a„,/3o) ^ W,/3o) weakly in L^(R4) x 

and ('Ua„./3o,wa„./3o) -t {ug^p„,vg^p„) a.e. in R^ x R^ as n oo. Since |AIa„| < 
+00 and is nonincreasing for n S N due to An —>■ +oo, by similar arguments as 
used in (14.321) and (14.431) . we can obtain that 

/ {a{x) + ^)"^ul^ p^Fb{x)vl^ p^dx^0 as noo, (4.45) 

Jr4 A„ 

where AIa„ is given by (12.21) . By the Fatou lemma, we have from (I4.45P that 
(ug,po,vg^/3o) £ (^a) X dl^(^b) with ug^p^ = 0 outside fla and z;o./3o = 0 

outside rife. It follows from the Sobolev embedding theorem, the condition (£> 2 ) 
and (I4.45|) once more that (ux„,PoTVXn,Po) {'ag,po,vg^pg) strongly in L^(R^) x 
L^(R^) as n —>■ 00 . Since iLg(r2a) x iLg(r2b) C Ex and (ma„,/3o: ^A„,/3o) is the 
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general ground state solution of (Pa„,/ 3 o) for all n G N, by the condition (D 3 ), 
it is easy to see that /a(“o,/ 3 o) = 0 and /^(r'o./So) = 0 in and 

respectively. Thanks to Lemma 13.111 and a similar argument as used in (j4.35ll , 
we must have vo,/3o = 0. Let ica„,/3o = ma„,/3o -uo,i3o- Then wa„,/ 3 o 0 strongly 

in and u>a„,/ 3 |j ^ 0 weakly in L^(R^) as n —>■ 00 . Furthermore, by the 

condition (II 3 ), the Brezis-Lieb lemma and [T71 Lemma 2.3], we also have 

JA„./3o(MA„,/3o,'aA„,/3o) = -/a„,/3o (wa„ ,/3o ,'aA„ ,/3o) + 4 (uo.^o ) + On (!)■ (4.46) 

Now, similar to (I4.35L we can get that mo,/ 3 o 7 ^ 0. Due to —oq ^ cr(—A, iJg (Da)) 
and the result of m, uo,^o is a least energy critical point of Ia{u). This together 
with ()4.46l) . implies (wa„,/ 3 o>'f^A„,/ 3 o) (OjO) strongly in L^(R^) x L^(K^) as 
n —>■ oo. It follows from the condition (Di) and the fact that (ma„,/ 3 o)' f^A„,/ 3 o) is 
a non-trivial solution of (Pa„,/ 3 o) that ||mo,/ 3 o lli 4 (R 4 ) ^ which is impossible. 
It remains to show that vx,i 3 ^ 0 for |/3| < 1 and A > A^. Indeed, if u\^p = 0, 
then by 5o > 0 and the condition (Di), we can see that c\^p > , which 

contradicts to Lemma 13.61 once more. | 

We close this section by 

Proof of Theorem 11.21 It follows immediately from Propositions 14.5114.61 


5 Concentration behaviors 

This section is devoted to the concentration behaviors of the ground state solu¬ 
tions and the general ground state solutions to (Pa./s) obtained by Theorems ll.il 
and 11.21 We first study such a property as A —-foo. 

Proof of Theorem 11.31 Let (ma„,/ 3 , I'An^a) be the solution obtained by The¬ 
orems o and O with A„ —^ -foo as n —>■ oo. Then by a similar argument 
as used in the proof of (3) to Proposition 14.61 we can see that p) —>■ 

G strongly in 

as n —>■ oo up to a subsequence if (ma^^s, v\„,p) is the ground state solution with 
/3 < /3o in the case of oq < &o < 0 or {ux„,p,vx^,p) is the general ground 
state solution in the case of ag < 0. Furthermore, (mo,/3, r'o,/?) is a solution 
of the equations (jl.lip . It follows from D[Jx,p(ux,p,vx,p)j = 0 in that 
{ux„,p,vx^,p) (uo,/ 3 ,"fo^s) in D^’^(]R^) x D^’^(]R^) as n —>■ oo up to a subse¬ 
quence. Thus, {ux„,p,vx„,p) —>• (mo,/ 3 , 'i^o,/ 3 ) in x iL^(R^) as n —>■ oo up 

to a subsequence. If {ux^,p,vx„,p) is the ground state solution with (3 < in 
the case of ag < bg < 0, then by Lemmas 13.121 and 13.131 {ug^p,Vg^p) is non¬ 
trivial. Thanks to lemma [3^ iug^ 0 ,vg^p) must be the ground state solution of 
(|l.llll . If (ma„,/ 3 , vx„,p) is the general ground state solution in the case of Og < 0, 
then by Lemma 13.81 once more, we have that either (mo,/3i'I' o,/?) is semi-trivial 
or {ug^p.vg^p) = (0,0). Note that Lemma [ST] holds, thus, it is easy to show 
that rnx„,p is nondecreasing for n and rnx„,p > 0 for all n G N. Therefore, we 
must have (mo,/ 3 , r'o,/?) is semi-trivial. Due to lemma [3781 we can also see that 


48 


















{uq^PjVq^Is) is a general ground state solution to (II.lip , which completes the 
proof. I 

We next study the concentration behaviors when /3 —>■ — oo. 

Proof of Theorem 11.41 Let (nA,,s„j W,/ 3 „) be the ground state solution of 
{'Px,/3rt) obtained by Theorem 11.11 with /3„ —>■ —oo as n —?► oo. By Lemma [3.81 
we have ||('ua,/ 3„,t'A,/3„)||A < 4(ma + mb) < S^. It follows from ()l.ip that 
{(■ua,, 3 „, W./Sn)} is bounded in E\ and H. Without loss of generality, we as¬ 
sume (ua./ 3 „,ua,/ 3 „) ^ (ma.ooWa.oo) weakly in Ex and U and (ua,/ 3 „,'Ca,/ 3 „) -t 
(ua.ooWa.oo) a.e. in x as n —>■ oo. For the sake of clarity, the following 
proof will be further performed by several steps. 

Step 1. We prove that (ux,i 3 ^,vx,p^) —t (ux,oo,vx,oo) strongly in 7^ as n —?► oo 
with ux,oo 0 and ua,oo ^ 0. Indeed, one of the following two cases must 
happen: 

(1) MA,oo = 0 or UA,oo = 0; 

( 2 ) uXjOo 7 ^ 0 and ua,oo 0 - 

If the Case (I) happens, then without loss of generality, we may assume ux,oo = 
0. By a similar argument as used in Proposition l4.21 we can see that ||ma,/ 3 „ ||^ 4 (r 4 ) 
> + o„(I). On the other hand, since /3„ < 0, there exists 0 < < I -|- o„(I) 

such that SnVx,pru £ Mj.a- Now, by Lemma 13.31 we have 

Jx,/3„{ux,p„,Vx,pJ > Jx,/3r,{ux,/3„,SnVx,l3j > -f TO 6 ,A + 0„(1). (5.1) 

Thanks to Lemma [4.61 mt^x mb as A —>■ oo. Thus, there exists A 3 > 0 such 
that (EU is impossible for A > A 3 due to Lemma 13.81 Therefore, we must 
have the Case (2). In this case, it is easily see from the Fatou’s lemma that 
/r 4 '“a 00 '^a 00 = b. On the other hand, since /3„ < 0, multiplying (Pa./ 3 „) with 
(ua.oo, vx,oo), we obtain that 

'Da,x{ux , 00 : '^X,oo ) < hA.oo||i4(M4) and Vb^xiyx ,00 ; '^AjOO ) < ll«A ,00 II■ 

Thanks to Lemma (3?^ there exists 0 < t* , s* < I such that (t*itA.oo, s*ua,oo) G 
A/a,/ 3 „ for all n, which, together with the Sobolev embedding theorem, implies 

T^\it*nUx ,cxD; ^n^A,oo ) > 'S>x{ux,p^,vx,p„) > 'Dxiux , 00 : ^A,CXD ) + o„(l). 

It follows from Lemma 1^751 and (11.11) that (wa,/ 3 „!'^A./Jn) (ma.oo; ^^a,oo) strongly 
in "H as n —>• 00 . 

Step 2. We prove that ua.oo G H^{{ux,oc > 0}) and z;a,oo G H^{{vx,oo > 0}). 

Indeed, since the conditions (iAi)-(L> 3 ) hold, ux,p^ and vx,p^ are both pos¬ 
itive in by the maximum principle and Lemma 13.51 It follows that ma,oo 
and vx,oc are both nonnegative in R^. Thanks to Step 1 and [S] Proposi¬ 
tions 3.8 and 3.9], {(nA,/ 3 „Wa,/ 3 „)} is also bounded in L°°(R^) x L“(R'^). By 
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[lU Theorem 1.7], {(VitA„a„, Vz;a,/ 3 „)} is bounded in T°°(R^) x i°“(R^). It 
follows from /3„ < 0 and the classical elliptic regularity theory (cf. [43l Corol¬ 
lary 3.5]) that ua,/ 3 „ and va,/ 3 „ are both local Lipschitz. Thus, 9 {ua,oo > 0} and 
9{fA.oo > 0} are both local Lipschitz and i7o({uA,oo > 0}) and i7o({r’A,oo > 0}) 
are both well defined. Furthermore, due to the extension theorem, we also have 
ux,oo G ^^o({ua.oo > 0}) and ua,oo S i7o({r’A,oo > 0}). Now, we can use similar 
arguments as used in the proof of [151 Theorem 1.4] (see also |49l Theorem 1.3]) 
to show the conclusions. | 

We close this section by 

Proof of Theorem 11.51 By Lemmas 13.81 and 13.91 we have rnx,j 3 G [m-a.A + 
ma+mt] for all A > Aq and ,5 < 0 in the case of qq < bo < 0. Furthermore, 
by Lemma [4.61 we have ma,x + 'mb,\ nria -f mb as A —>■ -l-oo. Thus, for every 
{(An, /3n)} satisfying A„ —>■ -l-oo and /?„ —>■ — oo as n —>■ oo, by a similar argument 
as used in Theorem 11.31 we can see that the ground state solution of (PA„„a„) 
obtained in Theorem 11.11 in the case of ao < 5o < 0 has the same concentration 
behaviors as described in Theorem 11.31 | 
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